So... firstly, I like the software. For the curious, my current configuration is composed of 4 x 4TB rotating storage as an 8TB RAID 10 array supplimented by 32G of RAM and 1TB of NVME (WD Black).
... so I'm reinstalling things on my new threadripper install ... and I'm enjoying the cache. The write-back behavior is only enabled on the NVME (not on the RAM) and the writes are piling up as the NVME is substantially faster than the drives. At one point it was over 100k blocks. This is expected as I'm gating in 1T or so of files (my old Steam directory) over the GigE network link.
... what I didn't expect was the whole show to stop-and-pause for minutes while the Deffered blocks went to zero. I can see that they might be throttled when they're hitting some maximum (although I didn't assign any part of the NVME for read vs. write) ... but 100% pause is _very_ disruptive. I suppose this won't happen too often, but still... it's not nice.
Deferred Writes that Pile up.
Re: Deferred Writes that Pile up.
Odd. I can't pull data over a GigE line faster than even a single 7200 RPM WD White spinner drive will write it at. And I'm fully saturating the network (at 900Mbps). What's the tested native write speed on your RAID 10 array, and is it writing these 1T of files sequentially or randomly?
Perhaps seeing your entire Primocache cache task configuration would help. It sounds like it's not flushing on a regular basis so that writes don't continue to pile up.
Perhaps seeing your entire Primocache cache task configuration would help. It sounds like it's not flushing on a regular basis so that writes don't continue to pile up.
Re: Deferred Writes that Pile up.
Well... the array is composed of WD greens... and the problem was occuring while writing a large terrabyte file to it.
... now regardless of that, now I'm reading from the terrabyte file (tar file) and extracting it. This runs up the defferred blocks _much_ faster. As it should ... but my point is that the hang for several minutes while the deferred blocks catch up behavior is bad.
... I would _expect_ the whole system to be slow... but not 100% stopped.
... now regardless of that, now I'm reading from the terrabyte file (tar file) and extracting it. This runs up the defferred blocks _much_ faster. As it should ... but my point is that the hang for several minutes while the deferred blocks catch up behavior is bad.
... I would _expect_ the whole system to be slow... but not 100% stopped.
Re: Deferred Writes that Pile up.
What you're describing I can't replicate.
Just created a cache task with a 4GB L2 cache on a SSD. Set it to read/write cache a large 500GB spinner drive (older, 5400 RPM), with deferred writes set to 60 seconds, intelligent deferred write mode, and shared read/write space. I then proceeded to copy over a 48GB file, which immediately filled up the L2. It kept copying at high speed however (TeraCopy still shows ~60-70 MB/s write speeds as the copy continues, down from ~120), and Urgent/Normal writes in Primocache keep going up as they should. Urgent are about twice what normal are, which you'd expect when the target volume is overloaded with work.
My test case never stalled or waited for the L2 to completely flush deferred blocks down to 0. That's why I think knowing your complete configuration details might help figure out what's going on for you. It's not a normal behavior of the L2 from what I've seen.
Just created a cache task with a 4GB L2 cache on a SSD. Set it to read/write cache a large 500GB spinner drive (older, 5400 RPM), with deferred writes set to 60 seconds, intelligent deferred write mode, and shared read/write space. I then proceeded to copy over a 48GB file, which immediately filled up the L2. It kept copying at high speed however (TeraCopy still shows ~60-70 MB/s write speeds as the copy continues, down from ~120), and Urgent/Normal writes in Primocache keep going up as they should. Urgent are about twice what normal are, which you'd expect when the target volume is overloaded with work.
My test case never stalled or waited for the L2 to completely flush deferred blocks down to 0. That's why I think knowing your complete configuration details might help figure out what's going on for you. It's not a normal behavior of the L2 from what I've seen.
Re: Deferred Writes that Pile up.
I did pretty much post my config, but here it is in full:
- ASUS ROG Zenith Extreme w/ Threadripper 1900X and 64G 3200 RAM.
- 4x WD Green 4T drives configured RAID 10 w/ AMD RAID 'C:' drive
- 2x 2T drives unused (For FreeBSD dual-boot)
- 2 port extra SATA board with
- 1x CDROM
- 1x 950 EVO 1TB (unused in the config).
- WD Black NVME (4x?) 1T on NVME daughterboard. This is not SATA... is't 4x NVME. Benches in the 3.5GB/s read/write.
Primo Cache config caches C: (8T) with 32G RAM and the 1T NVME. I've pasted a copy of the cache config panel.
- ASUS ROG Zenith Extreme w/ Threadripper 1900X and 64G 3200 RAM.
- 4x WD Green 4T drives configured RAID 10 w/ AMD RAID 'C:' drive
- 2x 2T drives unused (For FreeBSD dual-boot)
- 2 port extra SATA board with
- 1x CDROM
- 1x 950 EVO 1TB (unused in the config).
- WD Black NVME (4x?) 1T on NVME daughterboard. This is not SATA... is't 4x NVME. Benches in the 3.5GB/s read/write.
Primo Cache config caches C: (8T) with 32G RAM and the 1T NVME. I've pasted a copy of the cache config panel.
- Attachments
-
- Capture.PNG (26.54 KiB) Viewed 4974 times
Re: Deferred Writes that Pile up.
I just noticed, BTW, that I put these two questions in the wrong section --- They Belong in Primo Cache, not Primo Ramdisk.
Re: Deferred Writes that Pile up.
In the Advanced Level-2 Cache Options area (click the icon to the right of the L2 Size box), do you have individual read/write cache space setup? I disabled it for my test, but I think the default is to separate it when creating a new cache task.
In the Advanced Defer-Write Options area (click the icon to the right of the Defer-Write Latency box), what strategy is your cache task using? I normally go with Intelligent, since I find it more responsive to high activity than Native.
The only other difference between your scenario and mine, is that my test only used a L2 cache - I had no L1 defined for it. I'm wondering if the process of flushing from L1 to L2, and then L2 to the disk might be overloaded somehow. I've never seen it happen though - that's what the "urgent writes" in Primocache are for. When the cache is full/overloaded, it forces direct writes to the disk to keep the data flowing.
Perhaps try the first two option settings I posted and see if it makes a difference for you? Otherwise I think Support might have to chime in with suggestions.
In the Advanced Defer-Write Options area (click the icon to the right of the Defer-Write Latency box), what strategy is your cache task using? I normally go with Intelligent, since I find it more responsive to high activity than Native.
The only other difference between your scenario and mine, is that my test only used a L2 cache - I had no L1 defined for it. I'm wondering if the process of flushing from L1 to L2, and then L2 to the disk might be overloaded somehow. I've never seen it happen though - that's what the "urgent writes" in Primocache are for. When the cache is full/overloaded, it forces direct writes to the disk to keep the data flowing.
Perhaps try the first two option settings I posted and see if it makes a difference for you? Otherwise I think Support might have to chime in with suggestions.
Re: Deferred Writes that Pile up.
I'm including the two snapshots for you. But the short answer is that I have a combined read/write and I have "normal" selected.
- Attachments
-
- adv-def.PNG (7.79 KiB) Viewed 4954 times
-
- l2.PNG (11.89 KiB) Viewed 4954 times
Re: Deferred Writes that Pile up.
Perhaps try the Intelligent deferred write mode (or Buffer mode for your scenario) instead of Native. The different modes are detailed on this page, and I think using a slightly more aggressive mode would help. The native mode is just a "flush to disk every X seconds" strategy, which really isn't much of a strategy at all for a dynamic system.
I'll admit however, I'm still puzzled why yours wants to stop and flush the entire set of deferred writes to disk when it fills up. That's what urgent writes are for in Primocache.
I'll admit however, I'm still puzzled why yours wants to stop and flush the entire set of deferred writes to disk when it fills up. That's what urgent writes are for in Primocache.
Re: Deferred Writes that Pile up.
@zBeeble, yes, as Jaga said, usually you don't need to stop/pause cache, PrimoCache automatically does that.
If you often have large amount of data writing beyond write cache space, you may reduce the defer write latency to a smaller one like 5s or 2s. Alternatively you may use the "average" write-mode which is suitable for the scenario that has heavy write load and low throughput in target disks.
If you often have large amount of data writing beyond write cache space, you may reduce the defer write latency to a smaller one like 5s or 2s. Alternatively you may use the "average" write-mode which is suitable for the scenario that has heavy write load and low throughput in target disks.