Volumes and the Cache
A flexBlock volume is a block device defined by its superblock and chunk objects in the bucket, independent of any host. The cache device is local NVMe (or a partition or file) on the host that serves it.
Chunks
Section titled “Chunks”The unit of everything in flexBlock is the chunk: a fixed-size span of the device, 1, 2, 4 or 8 MiB, chosen at init and fixed for the volume’s life. The default is 2 MiB.
Each chunk is one object in the bucket and one slot on the cache device. The device advertises the chunk size as its optimal and maximum I/O size, so filesystems align to it without being told. See Choosing a Chunk Size.
What lives in the bucket
Section titled “What lives in the bucket”Everything under the grant’s prefix belongs to the volume.
| Key | Content |
|---|---|
<prefix>/<NN>/<chunk> | One chunk, compressed unless the volume uses none, overwritten in place by every flush; NN is the chunk number modulo 100 as two digits. A small header records the compression used, so changing the volume’s algorithm leaves older objects readable. |
<prefix>/__superblock | The volume’s definition: device size, chunk size, logical block size, compression, encryption setting, creation time and a random volume id. Independent of any host. |
<prefix>/__owner | Which host owns the volume: owner id, hostname, cache device, claim time, and whether it was taken over or released. See Ownership. |
Objects are uploaded with a checksum the store verifies (CRC32 on S3, CRC32C on GCS, MD5 on Blob and OCI) and verified again on download. On S3, SSE-S3 is requested by default.
The volume owns its prefix
Section titled “The volume owns its prefix”At every start the daemon rebuilds its index of which chunks exist by listing the 100 NN/ prefixes; nothing else lists the bucket while serving. That is why init refuses a prefix that holds any key other than leftover access or conditional-write probes: the volume must own its prefix outright.
What lives on the cache device
Section titled “What lives on the cache device”The cache device holds:
- Slots: chunk-sized regions that hold the cached chunks. Only this region scales with the cache size.
- Slot metadata: for each slot, which chunk it holds, whether it is dirty, and which pages are valid.
- A write-ahead log (WAL): the ring of dirty and discard intents that recovery replays after an unclean stop.
- Superblocks: the volume’s geometry, which bucket and prefix it belongs to, and whether the cache was stopped cleanly.
Everything except the slot data is small, fixed metadata. Together it lets a restarted daemon trust dirty slots and drop everything else after an unclean stop.
Clean and dirty
Section titled “Clean and dirty”A slot is dirty when it holds data not yet uploaded to object storage, and clean once the upload has landed. Clean slots can be dropped at any time, since the bucket has their data. Dirty slots exist only on the cache device until they are flushed, which is why commands that would discard them require an explicit flag, such as detach --allow-dirty-loss. See Where Data Can Be Lost.
Sizing the cache
Section titled “Sizing the cache”The cache can be much smaller than the volume.
- Its size determines how much of the volume can be hot at once, and how much dirty data can accumulate before writes wait for write-back.
- The minimum is one chunk plus about 332 KiB of metadata.
- A cache that is a meaningful fraction of the working set is what gives local-SSD performance.
When a volume moves to another host, only the volume’s geometry is carried across: the new host’s cache can be a different size and kind of device, and it starts empty and warms on demand.