Memory
The flexBlock daemon reserves most of its memory up front, when it starts. A preflight check estimates that reservation and refuses to start if it would crowd out the rest of the host. This page explains what uses the memory and how to bring it down.
Where the memory goes
Section titled “Where the memory goes”- I/O buffer pool. Queues × queue depth × chunk size, reserved up front when requests are copied through the daemon (copy mode). This is one of the large allocations.
- Staging buffer (zero copy). On Linux 6.16 or newer the kernel registers each request’s pages directly, and a per-queue staging buffer of the same size backs only misses and fallback writes. The preflight counts it in both modes.
- Transfer buffers. Buffers for uploads, downloads and deletes, which scale with the
--concurrent-*flags. These can outweigh the I/O buffer pool. - Object index. The in-memory set of chunks that have an object in the bucket. It grows as the volume fills and, on a large volume, can outweigh the buffer pool.
flexblock_object_index_entriesreports its size in chunks. - Slot metadata. Scales with the number of cache slots.
Page state under /run
Section titled “Page state under /run”The per-slot page bitmaps, about 70 MiB per TiB of cache, live in a root-only file under /run/flexblock so they outlive a daemon crash. /run is a tmpfs shared by every device on the host, and its size is set by your distribution, typically 10 to 20% of RAM. The daemon removes the file on a clean stop, detach or wipe.
If /run cannot hold the file with 5% of its size (at least 64 MiB) to spare, or the file is removed while the daemon serves, the daemon logs a warning and keeps the bitmaps in memory only. On a cache device with a volatile write cache, every write that adds pages then waits for its record, which slows small writes.
The preflight check
Section titled “The preflight check”At start the daemon compares a deliberately pessimistic estimate of its resident memory with MemAvailable:
- Above a quarter of available memory, it warns.
- When what it reserves at start exceeds half of available memory, it refuses with exit code 10, unless the cache holds dirty data or
--allow-overmemoryis given. The object index of a fully written volume only warns.
The refusal names the term that dominates the estimate. To run the check without serving:
sudo flexblock start /dev/nvme1n1p1 --dry-runShrinking the footprint
Section titled “Shrinking the footprint”Follow the term the refusal names:
| If this dominates | Do this |
|---|---|
| I/O buffers | Lower --nr-queues or --queue-depth, or use a smaller chunk size for new volumes. |
| Transfer buffers | Lower --concurrent-flush-reads, --concurrent-uploads or --concurrent-downloads. |
| Object index (warns only) | A larger chunk size for new volumes, or a smaller device. |
| Per-slot cache metadata | Use a smaller cache, or a larger chunk size for new volumes. |
Pass --allow-overmemory only if you have verified there is headroom.
| Flag | Default | Effect on memory |
|---|---|---|
--a1out-multiple <M> | 0.5 | Size of the 2Q ghost list as a multiple of cache slots, in [0.5, 64]. It holds chunk identities only, not data. |
--allow-overmemory | Off | Start even when the memory reserved at start exceeds half of available memory. |
--no-zero-copy | Off | Forces copy mode and its buffer pool even where the kernel could register request pages. |
--nr-queues <N> | One per physical core, at most 16 | Fewer queues shrink the copy-mode buffer pool. |
--queue-depth <N|auto> | auto: 64 for 1 MiB chunks, 32 above | Lower depth shrinks the pool in copy mode; cold sequential reads may lose some parallelism. |
The cache size itself also matters: slot metadata and the object index scale with the number of slots and with how much of the volume has been written.
How the daemon protects itself
Section titled “How the daemon protects itself”- OOM priority. At start the daemon sets its own
oom_score_adjto -900, so under memory pressure the kernel kills ordinary processes before the one holding the block device. If the host denies the write, the daemon logs a warning and serves anyway. - I/O flusher. The daemon marks itself an I/O flusher, so memory reclaim in the daemon never waits on writeback to the device it serves. Without
CAP_SYS_RESOURCEit logs a warning and serves anyway.
Monitoring memory
Section titled “Monitoring memory”The flexblock_jemalloc_*_bytes series report the daemon’s allocator view of memory: allocated, active, mapped and resident bytes. See the Metrics Reference.