The case for smaller block size

[Davey126]

This may have been covered elsewhere and/or perhaps obvious to some. But for me it was an 'ah-ha' experience. Thought I would share and see what others have to say.

I use PrimoCache in a desktop environment utilizing small L1 caches (typically 256MB-1GB) to accelerate writes on traditional HDDs and a few older SSDs. SSDs also benefit from a modest reduction in write amplification as evidenced by the number of trimmed blocks using a 10s write deferral. A couple machines still have a 32-bit OS installed with 4GB mem and utilize 'invisible memory' for the cache. Works great and leverages an otherwise unused resource.

Initially I went with 32KB block size to reduce CPU/memory overhead given modest resources on the target machines. On a typical day I observed a 5-8% reduction in media writes due to trimmed blocks. I was pretty happy with that 'bonus' given all other benefits associated with PrimoCache.

A few days ago I inadvertently configured a read/write cache with 8KB block size. I soon noticed an up tick in the number of trimmed blocks (logical given the smaller size) along with a significant drop in the percentage of data actually written to disk. At present I'm seeing a 18-20% difference between normal and total writes. Amazing!

Looking for a downside I started monitoring CPU usage for various processes. No significant gain with 8KB blocks given the modest workload in a typical desktop environment. The overhead hit was also modest given the relatively small size of the disks being monitored.

In short, using a 4KB-8KB block size for smaller SSDs may prove beneficial provided the workload is modest and CPU/memory resources are not constrained.

[gabrielmorrow]

one thing to note is you should also not set it below the size of the filesystem cluster size

[Davey126]

Quick follow-up. I have been using the configuration outlined in the original post for several weeks, tweaking various parameters on occasion and observing impact. I eventually settled on a 16KB block size which routinely yields an 8-10% reduction in media writes. Of course, this is petty stuff in the grand scheme of PrimoCache benefits but thought I might share in case others have a similar interest/need.

[Davey126]

Yet another follow-up. I recently increased write deferral time to 30s to what (if any) impact it would have on overall system performance. Even with a UPS the longer timeframe makes me a little nervous as an unexpected crash could leave a fair bit of data unwritten. As expected there was little perceived gain in responsiveness or any other subjective measure as the underlying SSD is pretty speedy for the type of loads I throw at it. However, there was a HUGE impact on the amount of data actually written over the course of a day. While I previously saw a 15-20% delta between normal (+urgent) and total writes with a 10s deferral that number nearly doubled with a 30 second deferral. I often see deltas approaching 40% which is truly amazing. Trimmed blocks typically account for about 20% of the reduction. The remaining 80% amounts to duplicate writes that the L1 cache automatically discards. Over the course of 24 hours this amounts to multiple GBs of data that are never written to the SSD.

My system and PrimoCache settings are quite modest: An old Core 2 rig running Win8.1 x64, 4GB memory, 128GB system drive, 512MB L1, 16KB block size, no L2.

[Bjameson]

Thanks for sharing your findings. I have also been experimenting in the past and came to roughly the same conclusions. In theory small blocks are costly, both in memory and in CPU cycles. Yet on a fairly modern system it makes no difference. PrimoCache appears to use very few CPU cycles and I can easily spare the memory overhead of small blocks. So I use 4K blocks.

Also, I use a 5-minute write delay. This results in a higher hit rate and huge numbers of trimmed blocks. Especially while (de)installing programs or compiling large projects, as I often do. It's a clear win-win scenario.

As for the risk of data loss: while copying large files (VM disks) I observed that Win 8.1 x-64 uses a large write buffer. When the copy target is slow and W8 has the memory, it will buffer 10 Gigabytes or more to memory (Windows memory, not Primocache memory). It can take several minutes to flush its buffer. When copying to USB 2.0 it can take 5, 10 minutes to flush. In the meanwhile, data still in memory is at risk. Obviously Microsoft sees no problem with that. So using a 5-minute delay for Primo should also not be a problem. If the power goes you have damage, with or without Primocache. Data safety largely depends not on the system itself, but on the guys outside cutting cables with their digging machines.

As for UPS: in the '80s the company I worked for sold large numbers of UPS. Customers returned them in equally large numbers because of blown converter transistors. They could blow up in your face any time. So a UPS is no guarantee for protection. Yet a fully charged laptop battery will give you nearly 100% protection. So again, data safety is relative. I think PrimoCache adds little risk to the overall safety picture.

As long as you frequently backup your important data, the gain in speed often outweighs the risk of a disaster. So do, do experiment.

[Davey126]

Great observations. I was aware recent versions of Windows cached large writes to slow devices (assuming sufficient memory) as evidenced by Windows declaring a copy operation complete while the drive light might continue flickering for 30 sec or more. I wonder how long Win 7/8.1 will allow 'system' data to remain cached before executing a flush? I suspect it's more aggressive but there is still a period of vulnerability. In my case I have few worries beyond the potential annoyance of a misbehaving system. I catch a full system image weekly with daily incrementals. Has saved my butt on more than a few occasions.

With your comments in mind I may nudge the write delay up a bit to see what benefits materialize. At this point is more of an academic exercise as my system rarely lags unless I exceed the raw capacity of the ancient MB/CPU.

[Stubi]

I have deferred writes at 1 hour. If I create some important data I flush them to disk immediately. Or I pause deferred writes for a while for instance at huge writes so that I do not create urgent writes. Same I do before I create backups. Perhaps a feature to customize certain file types and folders that will be written immediately would be helpful.

One day I had a great experience. I installed a huge software that I did not like afterwards. After a power loss and a restart I had a clean system again without the install. Perhaps a new feature for PrimoCache

[Bjameson]

@Davey126: Windows will always write critical system data without delay. It does so by setting the 'FILE_FLAG_NO_BUFFERING' and/or the 'FILE_FLAG_WRITE_THROUGH' flags during critical operations. This ensures that data is physically written ASAP. Third-party applications can do this too, if necessary. Caches such as PrimoCache are expected to honour those flags by immediately pushing the data forward through the cache so that it hits the disk instantly.

However, I suspect (actually I'm quite sure) that Primo ignores those flags. No big deal, provided all writes stay sequentially ordered in the cache so that -for example- $MFT is updated at exactly the right moment. It's a questionable technique to do this but I'm not complaining because it makes the whole thing faster. The only disadvantage is an increased risk of massive damage once the Fortran hits the fan.

@Stubi: exactly for those reasons you received a free uninstall. But believe me, you were lucky this time

[Davey126]

@Bjameson: Thanks for the clarifications. In the end it may not matter for certain operations. Elsewhere in this forum someone asked about the interrelationship between the Windows and Primocache caches and which takes precedent. While they both may contain similar data (as explained in those posts) the Windows read cache definitely gets hit first on my system. Assuming the write cache behaves the same way a 'critical' system action might populate both caches but Windows should immediately commit the data from its own cache. I have no hard evidence to support this theory (and Stubi's experience suggests otherwise) but I'm pretty sure it would work this way.

[Stubi]

@Stubi: exactly for those reasons you received a free uninstall. But believe me, you were lucky this time

I was not lucky THIS time. I would be lucky at any time as long as the deferred write is not written to disk. I have been using PrimoCache on a RAID 0 system with many external disks and many TB of data (about 15 disks with about 11 TB). Only 1 time I had a problem after a system crash that the file index was not written back correctly. But this was a at a very early version. Since then I could not find one problem. I think it is pretty stable and does what it is designed for - but I have to admit that I never used the L2 cache. I was happy for the uninstall even if I always use software to create a trace to uninstall everything. But they are still not as perfect as a long deferred write time Sure - I do not know if it is possible since PrimoCache is not at the file level - customizing file types or folders to skip deferred writes would be helpful.