Caching a portable drive

FAQ, getting help, user experience about PrimoCache
Post Reply
mattj
Level 1
Level 1
Posts: 3
Joined: Fri Nov 09, 2018 12:38 pm

Caching a portable drive

Post by mattj »

Hello,

I am relatively new to PrimoCache, so I am seeking your expert opinions on a solution I am trying to implement.

I have a desktop and a laptop, each with internal SSDs. I only use one system at any given time - the laptop is for when I'm working away from my desk.

I like being able to pick up my laptop and have all my data with me. That is, *all* my data - documents, photos, Hyper-V machines, etc.

So, normally, I would:
• Keep my current, frequently-used, or performance-critical files on the internal SSDs. Everything else lives on a portable USB HDD that I switch between systems as needed.
• Sync the internal SSD data between the laptop and desktop somehow (e.g. OneDrive, scripts/utilities, or god forbid, manually). And this is a particular pain with large Hyper-V files which are typically locked by the host.
• Manually demote non-current data to the portable drive to free up internal space.

I want a better solution, and my idea is to store *all* my data on the external drive and use a very large L2 cache on the internal SSDs to bring the performance levels back up.

My goals are:
Simplicity: Data all in one place, rather than split between internal and external drives. No need to manually demote data out to the external drive to free up space on the internal SSD. And a simpler backup profile as a bonus.
Convenience: No need to go through the pain and delays of syncing data whenever I want to switch to the laptop or back again. Or having to remember to turn on the laptop in good time to allow background syncing to finish before I leave the house.
Efficiency: No need for the constant CPU/bandwidth overhead of running a sync client like OneDrive (especially when I only switch systems infrequently - most of the time we are pointlessly re-syncing).
Concurrency... Is NOT a goal. I have no desire or expectation to be able to access the data on both systems at the same time.
Performance: I obviously cannot work at portable HDD speeds, hence the need for PrimoCache.

But, I have read all about offline modification. And offline modification is kind of the crux of this idea. It will, of course, occur every time I move the drive between the two machines.

So, on to the practical considerations:

I am interested in the "Volatile" setting applicable to the L2 cache. My understanding is that if the L2 cache is marked "volatile", then it is discarded on reboot. This would seem to allow "offline modification". Great...

...But I reboot far more frequently than I switch between machines, and there is no way that losing the entire L2 cache on reboot is going to be acceptable.

I was optimistic when I saw the "Gather" setting. Having read about it, I figured, great, PrimoCache has a strategy for automatically loading up my L2 cache in the background. That would reduce the pain of having a volatile L2 cache that is wiped on reboot. (I assumed this was the L2 equivalent of the L1 "Prefetch" setting.)

But I have tested, and I'm not sure what "gathering" is supposed to be happening, because my L2 cache remains empty even after a long idle period after boot. And changing the Gather setting has no effect either. Am I missing something here?

So, it would seem I have to set a non-volatile cache, and *explicitly remember* to reset the cache if I move the drive. And if ever I forget, I risk data corruption?! That doesn't sound acceptable either..

I could imagine crafting a little program that would display a prompt upon Windows shutdown to ask if I am going to unplug the portable drive and, if so, command PrimoCache to reset the cache. I haven't yet investigated whether that's feasible. Or, a little sticky tag on my USB cable ("REMEMBER PRIMOCACHE!") Yeah, you can see the "meh" factor.

So, I'm very curious to hear your thoughts, and I'm open to suggestions if I seem to be on the wrong track with this idea!
User avatar
Support
Support Team
Support Team
Posts: 3623
Joined: Sun Dec 21, 2008 2:42 am

Re: Caching a portable drive

Post by Support »

This is a good usage scenario! But I'm afraid that it is kind of difficult to get a perfect solution.
L2 "gathering" is not worked as "prefetch". So you may just forget this option. A possible solution is to write a script for ejecting external HDD drive. In this script, you set "Volatile L2" option for the cache task, and eject the external drive. Each time you need to remove the external drive from a computer, you run this script first. So next time when you use this computer and plug in the drive, L2 cache will be automatically cleared and won't encounter problems. You also need a script to remove "Volatile L2" when you come back to use this computer with external drive. Without "Volatile L2" option, L2 cache content will not be cleared after reboot.

PrimoCache supports Command-line interface (CLI), you may easily use it into a batch script. Regarding the CLI, please see
http://www.romexsoftware.com/en-us/prim ... rface.html

Hope it helps.
mattj
Level 1
Level 1
Posts: 3
Joined: Fri Nov 09, 2018 12:38 pm

Re: Caching a portable drive

Post by mattj »

Thanks very much for your quick reply!

So I understand that the volatile option should normally be off, but could be set via a script when the drive may be removed. Thank you - I will explore these possibilities.

Please can you tell me what the L2 gathering is, so that I can have a better understanding of the software? I could not find a description of it in the documentation.

I have a related question about Defer-Write. I will need it, so that I am not limited to the write speed of the portable drive. But I would like to understand exactly what will happen if there is a system hang/BSOD/power failure.

I planned to reduce the risk by either disabling L1 cache or setting it to 100% read.

However, for L2, given that L2 cache is persistent, I would assume that in a system crash scenario, any deferred writes in L2 cache should be preserved and then flushed when the system reboots.

But from reading the documentation, I worry that PrimoCache would normally delete L2 cache after a system crash. Please could you explain what actually happens to deferred writes in a non-volatile L2 cache when the system reboots after a crash?

Is the "Ignore Sync Check..." option relevant in this case?
User avatar
Support
Support Team
Support Team
Posts: 3623
Joined: Sun Dec 21, 2008 2:42 am

Re: Caching a portable drive

Post by Support »

The "L2 gathering" option in the advanced L2 settings is to adjust the speed of saving data from source disk to L2 cache when computer is in busy state, in order not to affect/slowdown normal computer tasks. When computer is in idle state, this option has no effect and PrimoCache gathers L2 cache data as quick as possible.

Regarding the defer-write, so far the risk is same whether you use L1 or L2. Even you use L2 only, in a system crash scenario, no deferred writes in L2 will be flushed at reboots. As you have already known, PrimoCache will normally reset L2 cache after a system crash. We're working to support write-back on crash, but I'm not sure when it will be implemented.
mattj
Level 1
Level 1
Posts: 3
Joined: Fri Nov 09, 2018 12:38 pm

Re: Caching a portable drive

Post by mattj »

Thanks again.

So now I understand gathering to be the process of writing data to the L2 cache after it has been read from the source disk. I can see how it makes sense to have the ability to delay this activity based on system load.

Perhaps L2 prefetch would be a nice new feature, to automatically preload L2 after a reset, based on the most frequently accessed blocks from before the reset. (I assume you can still preserve the index of frequently used blocks, I guess it doesn't matter if the contents changed because preload would only be a "best effort" anyway.)

In my situation, I will probably wait for the write-back on crash feature you mentioned, as the possibility of data loss, although unlikely, would incur too much risk for me (Hyper-V machines).

Cheers and all the best!
Post Reply