How flexBlock Works

If you understand the pieces on this page, the volume in the bucket, the cache on NVMe and the daemon between them, you can predict what every flexblock command will do and why it sometimes refuses.

Architecture

Applications and a filesystem use /dev/ublkbN; the flexBlock daemon serves it from an NVMe cache and keeps the durable copy in an object storage bucket

Three things take part in serving a volume:

The kernel exposes the block device, /dev/ublkbN, through its ublk driver and hands each request to userspace.
The daemon, flexblock start, answers those requests from the cache where it can and from object storage where it must, and streams dirty data back to the bucket.
The cache device holds the chunks in use, together with the metadata that makes a crash survivable.

The durable copy of the volume always lives in the object storage bucket. Reads and writes are served from NVMe; the bucket holds the durable copy, compressed unless the volume uses none. See Volumes and the Cache for what lives where.

The data path

Reads

A read of data that is already in the cache is served straight from NVMe.

A read of data that is not in the cache is a miss. The daemon fetches the chunk’s object from object storage, decompresses and verifies it, stores it in the cache, and returns the requested range. Later reads of that chunk hit NVMe.

When the daemon sees a sequential cold read, it prefetches the next chunks in the background and widens that window as the scan continues, up to --read-prefetch. A scan of cold data is therefore not paced by one round trip to object storage per chunk, while a random read hydrates only a few extra chunks.

Writes

A write is durable on NVMe before it is acknowledged. The upload to object storage happens later, in the background.

When a write first makes a chunk dirty (holding data not yet in the bucket), the daemon also records a dirty intent in a write-ahead log on the cache device. The log records intent, not data; the data is in the cache slot. After a crash, that log tells the restarted daemon which chunks still need to be uploaded.

Flush

A background task uploads dirty chunks to the bucket. It flushes chunks that have been dirty longer than --flush-threshold (4 minutes by default). Each flush compresses the chunk with the volume’s algorithm, if any, and overwrites the chunk’s one key in place, and the chunk becomes clean once the upload has landed.

When the cache fills with dirty data, flexBlock works harder to drain it:

From the moment the dirty share of the cache reaches --max-dirty-ratio until it falls below --target-dirty-ratio, the daemon flushes aggressively regardless of age.
Over the same span, writes that would dirty a chunk that is clean or not cached are paced to the rate the bucket is absorbing data. Writes to chunks that are already dirty are never delayed, so a database’s hot pages, journal and log keep their latency.
Upload concurrency rises under pressure and backs off when object storage starts retrying requests or slowing down.

If uploads stop altogether, such writes slow to a crawl rather than fail. See Write-Back Behavior for the flags involved.

Discards

DISCARD and WRITE_ZEROES from the filesystem mark the covered pages as zero, so reads return zeros. A chunk whose pages are all zero has its object deleted instead of uploaded, or nothing at all if it never reached object storage. Failed deletes are retried every 30 seconds.

Whether you mount with discard or run a periodic fstrim affects how freed space reaches the bucket; see Filesystem Settings.

Shutdown

On SIGINT, SIGTERM or SIGHUP the daemon:

Refuses the signal while the device or one of its partitions is in use (mounted in any namespace, used as swap, or claimed by device-mapper or another kernel user), and logs the user instead.
Gives a process that merely holds the device open five seconds to close it, or until a second signal, after which it sees I/O errors.
Flushes every dirty chunk and finishes pending discard deletes within --shutdown-flush-timeout (one hour by default).
Checkpoints and marks the cache device cleanly stopped.

Dirty data not uploaded in time stays on NVMe and is flushed by the next start. Once the stop is under way a second signal is only logged, and a third exits at once, leaving the cache unclean for the next start to recover. See Stopping Cleanly.

Recovery

If the daemon dies, the kernel keeps the block device in place rather than failing its I/O, so a mounted filesystem blocks until start runs again.

After an unclean stop, start keeps the dirty chunks, drops everything else it cannot trust, and replays the write-ahead log to find dirty chunks whose flush never completed. It then recovers the existing /dev/ublkbN, the kernel reissues the requests that were in flight, and the filesystem carries on. See Recovering From a Crash.

Where data can be lost

Dirty chunks exist only on the cache device until they are flushed. A failed cache device is the one hardware event that costs acknowledged writes: at most what flexblock_cache_oldest_dirty_age_seconds reported at the time, though what remains in the bucket may need a filesystem check. Everything else in flexBlock is designed to refuse rather than lose data, apart from a few commands that require an explicit flag. See Where Data Can Be Lost.