Romex Software Forum

support wrote: ↑Wed Oct 31, 2018 7:08 amNow, I see you have a spare 2TB ssd raid. Yes, it is able to use it as write cache for the HDD. Jaga suggest using L2 only, but I suggest L1 + L2 as RAM has a better write performance than SSD (raid).

I just wasn't totally sure if when the L1 filled up with write requests, if it would "spill over" to the L2, or go directly into urgent writes instead. So I didn't suggest both for that reason. If the L2 is used when the L1 is overwhelmed with write requests, then absolutely use both.

Yes, L2 will be used to buffer incoming write-data when L1 is full though L1 will do urgent writes to target disk.
viewtopic.php?f=33&t=4632#p12661

If you're using the Seagate Reds for backup only, then there is absolutely no point in caching them at all - you are not going to be repeating any reads or writes and you do increase (albeit slightly) the probability of data loss in the event of a system crash or power failure.

The one suggestion I would make is to put your backup drives in external enclosures and connect these via e-SATA or USB 3.0+ in order to make it easier to change drives. An NAS cabinet is another option to consider though you'll want it connected via gigabit Ethernet at the very least (and 10GbE or multi-link GbE would be better since you'd presumably want 500+MB/s write speeds) since that can add RAID remote storage,.

You may want to read this thread where I did a lot of testing in moving a mixed test group of primarily video files with various cache settings.

@ zeroibis - which thread is that? Thanks.

Oh sorry about that: https://community.covecube.com/index.ph ... ment-26769

Hey,

that is pretty similar to what I am doing here, film post production on large 64 TB RAIDs and using disk to disk backup for daily nearline backup purpose.

As Support and Jaga already mentioned, a read cache doesn't really help on your backup need, because you will likely only read the data once from the source disk. On the target disk side, L1 RAM + L2 SSD RAID + read&write cache + defer writes is highly recommended. And its not only the defer write that is speeding up the process, but the fact that the MFT of your target disk is held mostly in the write cache. If you transfer many small files (image sequences for instance...) then you will usually see a huge performance hit in copy operations, due to the fact the HDD read/write heads are flying back and forth between where the new data is written and where the MFT sectors are placed and where the MFT backup is placed (usually 1/2 disk size away...). In turn that leads to heavy load on head seeks and slows down HDD writes significantly. By using PrimoCache you can optimize that behavior strongly.

I just recently made a disk to disk copy from one 64 TB partition to another one using that method.

Btw, I highly recommend using software such as "Syncovery" for this backup process. It can handle incremental backups, versioning and all such neat things you need and it will easily handle really large datasets. You can even offload to cloud if needed/desired.

Using Syncovery you can create many individual backup tasks with very specific settings. You can let these run e.g. every night to make automatic backups of your current projects, including versioning of overwriting files.

Beside that do you create catalogs from your disk to allow offline search for files?
We use "Broken Cross Disk Manager" for this. It creates inventories of your disks, including contents within compress zip, 7z, rar etc. files plus it can handle LTO LTFS to catalog them as well (which is why we use it). Its not a MAM solution, but it works like a charm for me and is really low budget.

Do you regularly defragment your source disk?
We defragment our 64 TB partitions using space defrag and we usually on see less than 10 fragmented files during the day. This makes these volumes/partitions behave significantly faster. You may employ caching your source disk as well for this, as defragmentation speed is significantly better with PrimoCache enabled. However, your risk is always that write cache is currently at risk if you experience an unexpected crash.

Just a few ideas...

Cheers,
Axel

I should add the wish for PrimoCache to allow read-ahead caching in that context.

When data is sequential on defragmented files, it may improve read speed a lot if PrimoCache makes sure to preload a custom number of blocks BEYOND the last requested block.

Why?

Because whenever the next streaming block is requested it is then very likely it can be fed from the cache instead of waiting for disks to spin around.

I mentioned that before and I am quite sure it would help in this scenario too.

Cheers
Axel

Axel Mertes wrote: ↑Thu Nov 29, 2018 4:54 pm I should add the wish for PrimoCache to allow read-ahead caching in that context.

Sure, read-ahead is under our consideration.

support wrote: ↑Fri Nov 30, 2018 6:40 am

Axel Mertes wrote: ↑Thu Nov 29, 2018 4:54 pm I should add the wish for PrimoCache to allow read-ahead caching in that context.

Sure, read-ahead is under our consideration.

SUPER