Question about raw block device vs. file based disk cache storage on modern hardware

[ksqrtr]

Hello!

Is it still recommended to use raw block device for disk cache storage on modern hardware with good NVME disks? How it would compare in terms of overall performance with files' based cache storage on say ZFS (or XFS/EXT4)?

[cmcfarlen]

Hello,
It's an interesting question. There is some discussion/justification on the chosen architecture for cache in the docs here: https://docs.trafficserver.apache.org/developer-guide/cache-architecture/architecture.en.html#cyclone-buffer

The disk layout is light on structure and uses the circular nature to avoid a lot of bookkeeping that a filesystem approach would need.

[ksqrtr]

Hello, It's an interesting question. There is some discussion/justification on the chosen architecture for cache in the docs here: https://docs.trafficserver.apache.org/developer-guide/cache-architecture/architecture.en.html#cyclone-buffer

The disk layout is light on structure and uses the circular nature to avoid a lot of bookkeeping that a filesystem approach would need.

My initial question is a bit more simple, as I've meant current ATS cyclic cache architecture but made upon file(s) (say one huge cache.db file), not raw block device(s). As far as I understood from reading docs and issues here general recommendation was to use raw block device(s), but may be with modern hardware and filesystems there will be no tangible difference with files' based cyclic cache?

[spijet]

If I may add to the initial question:

Does it make sense to use a logical block device situated on top of some volume manager (e.g. MD RAID, LVM or ZFS ZVOLs) over raw disks? In case of MD/LVM we'd get redundancy and/or performance benefits (since the virtual block device can live on an array of physical drives), and in case of ZFS we can reap the benefits of ZFS's adaptive block cache (ARC) which can cache hot data in RAM in MRU and MFU (Most Frequently Used) fashion, while also having data compression and checksumming.

[bryancall]

We had a discussion about this in our weekly PR/issue scrub and we feel there will be a small overhead in storage space and possibly performance when using a filesystem and file vs block device. Since no one on the call has actually benchmarked it we don't know how much the overhead would be.

My guess is it would be very small and hard to measure.