Page 2 of 3

Re: L2 Cache Not Used When Needed

Posted: Mon Jan 07, 2019 6:10 pm
by mell111
The Urgent writes definitely increased substantially, but I can't say that it was steady (as opposed to bursts) since I wasn't monitoring the whole time.

Re: L2 Cache Not Used When Needed

Posted: Tue Jan 08, 2019 7:20 am
by Support
Not sure if this is the cause, but if possible, you may try a smaller latency like 10s, 30s or 60s to check if the problem still happens or not.

Re: L2 Cache Not Used When Needed

Posted: Tue Jan 08, 2019 6:25 pm
by mell111
I tried that - it didn't help. The longer latency appears to have reduced the incidence of this (though it could be a coincidence), but once the thrashing begins, the only thing that works is to stop app activity to the affected disk and allow PrimoCache to catch up. I find it puzzling that even with this stopped activity, PrimoCache still takes a few minutes to write about 2GB, but I presume its because of internal activity of the drive.

Can you tell me what to look for if I encounter this again? Also, as it stands now, should PrimoCache reduce its writing activity when the underlying disk is very slow and it has cache space available? If not, what do you think about that? Thanks.

Re: L2 Cache Not Used When Needed

Posted: Wed Jan 09, 2019 5:51 am
by Support
We'll introduce a new writing option to coordinate the work of L1 and L2 write cache. It might be helpful to this problem.

Re: L2 Cache Not Used When Needed

Posted: Wed Jan 16, 2019 2:04 am
by mell111
I have an additional data point that illustrates the problem with more clarity. The screenshot below shows no writing to L2 cache and plenty of available L1 cache and hardly any urgent writes while the underlying drive was thrashing and the writing app was slowed blocking on writes (indicating, I think, that PrimoCache is passing along the writes directly to the drive, rather than caching). When the deferred blocks reached around 20,000 I suspended the app and PrimoCache slowly recovered as before. This whole episode lasted around an hour with the thrashing probably lasting around 10 minutes. Is this expected behavior under the circumstances? Can you say a few words about the new writing option and how it might help? Thanks.
PrimoCache-20190115.png
PrimoCache-20190115.png (16.92 KiB) Viewed 4054 times

Re: L2 Cache Not Used When Needed

Posted: Wed Jan 16, 2019 5:24 am
by Jaga
Just an observation based on that screenshot - your Urgent Writes are a little over 100MB, while total writes are over 86GB. Do you ever get into a scenario where urgent writes are a significant portion (i.e. Primocache truly "spills over" and hammers the drive directly with urgents)?

Re: L2 Cache Not Used When Needed

Posted: Wed Jan 16, 2019 6:37 am
by mell111
Typically in my usage urgent writes are less than 3% or %4. It was a coincidence that the scenario I described in this thread came up shortly after I started a session, so the data in the screenshot is not mixed with a lot of "normal" data (although there was no thrashing for approximately 30 minutes from the start). The odd thing to me is that PrimoCache is not caching incoming data as it struggles with slow writes to a thrashing drive (under normal circumstances it does continue to cache as it write to the disk at normal speed.) If it did, this should allow the drive to "catch up" and recover, but being hammered by both PrimoCache and the app doesn't allow this. Also, on two occasions my system crashed almost certainly in this scenario (I can't be 100% sure since it didn't happen while I was observing, but my system has been rock solid for a couple of years now.)

Re: L2 Cache Not Used When Needed

Posted: Mon Jan 21, 2019 6:40 am
by Support
@mell111, yes, the data in the screenshot is kind of unexpected as I see Total Write on L2 is 0. This indicates that L2 didn't cache incoming write-data when L1 is full. Does your writing app bypass volumes and directly write data to the underlying disk, like write-through mode?
The new write option we plan to add is to remove L1 urgent flushing when L2 has plenty of write space. That is, during the latency time, L2 will continue to cache incoming write-data when L1 cache is full, and L1 will not start flushing its deferred data to disk.

Re: L2 Cache Not Used When Needed

Posted: Mon Jan 21, 2019 9:17 am
by mell111
No - my writing app does nothing unusual, it just does plain writes. I can confirm from many observations with Windows Task Manager that under normal circumstances (no thrashing of the underlying disk) when the given volume is cached, only PrimoCache writes to it at the approrpriate interval.

Also, it isn't just L2 cache that isn't getting written to - there is plenty of available space in L1 (about 75% by my calculation) and it isn't utilized either.

Thanks for clarifying the new write option. Given that in the above screenshot there is hardly any urgent writing, what do you think would happen? If the normal writing will continue (at the latency interval) then the disk will continue to thrash. Do you think it's reasonable to have an option to check write perormance at the latency interval and continue to defer writes when there is plenty of cache space available and disk write performance is very bad (in this case I'm getting much less than 1 MBps with occasional spikes to 2-3 MBps.) Even very slow disks don't perform that badly unless they're thrashing.

Thanks.

Re: L2 Cache Not Used When Needed

Posted: Tue Jan 22, 2019 1:54 am
by Support
Did you check if other apps will cause such problem? For eg, when L1 is flushing, you copy files to the volume cached by L1.
If this problem only happens with that writing app, is it possible that you send us this app for our analysis?