Common Issues

FlexBlock prefers to refuse rather than guess. Nearly every refusal names the state it found, the state it needed, and the flag that would override it, and exits with a code that says whether a retry can help (see Exit Codes). This page maps the common symptoms to their causes and fixes.

Hangs and stops

A mounted filesystem hangs and umount cannot be interrupted

Cause: The daemon died. The kernel keeps the device quiesced and blocks its I/O until a daemon returns.

Fix: Run start on the same cache device. It recovers the device and the blocked I/O drains.

sudo flexblock start /dev/nvme1n1p1

systemctl stop of the flexBlock service hangs

Cause: The daemon received the stop signal while the device was still in use (mounted, perhaps in another mount namespace, or claimed by another layer) and refused it. The journal shows device is still in use.

Fix: Unmount or release the device by hand, then send the signal again:

sudo systemctl kill -s SIGTERM flexblock@nvme1n1p1.service

Otherwise systemd waits for TimeoutStopSec and then sends SIGKILL, which leaves the cache unclean for the next start to recover.

top shows about 50% iowait while the volume is idle

Cause: The kernel is older than 6.15 and lacks IORING_ENTER_NO_IOWAIT, so it counts each idle flexBlock queue thread’s wait as iowait. On a 16-vCPU host with 8 queues that is exactly half the CPUs. The start log says the flag is unavailable.

Fix: Nothing is waiting on storage; this is an accounting artifact. Use iostat -x on the cache and ublk devices for real utilization, and discount the queue threads’ CPUs in mpstat (the start log names them). On Linux 6.15 or newer the idle threads are ordinary sleepers.

start refusals

start refuses with exit code 12, busy

Cause: Another daemon is already serving this cache device.

Fix: Find it with flexblock show, which reports the PID. Stop it with SIGINT if it should not be running.

start refuses because a leftover ublk device has different geometry

Cause: A stale kernel device from a previous daemon or a previous volume.

Fix: Remove it with flexblock reap <id>, then start again. Or pass --replace-ublk if the point is to change geometry. See Leaked Kernel Devices.

start refuses because a leftover ublk device lacks a volatile write cache (exit code 9)

Cause: The leftover device was created when completed writes were durable. This run’s are not, because of --volatile-write-cache, --cache-io-mode buffered, or a cache device that now reports a volatile write cache or lacks O_DIRECT support. Recovery keeps the device’s original attributes, so the kernel would never send FLUSH.

Fix: Drop --volatile-write-cache or --cache-io-mode buffered if either was added. Otherwise pass --replace-ublk, which forfeits the reissue of in-flight requests.

start refuses on memory (exit code 10)

Cause: The memory start reserves exceeds half of MemAvailable.

Fix: Follow the remedy the refusal names: lower --nr-queues or --queue-depth (or use a smaller chunk size for new volumes) when I/O buffers dominate; lower --concurrent-flush-reads, --concurrent-uploads or --concurrent-downloads when transfer buffers do; use a smaller cache or a larger chunk size for new volumes when per-slot metadata does. A large object index only warns; a larger chunk size for new volumes, or a smaller device, shrinks it. Pass --allow-overmemory only if you have verified there is headroom. See Memory.

start refuses with exit code 9: the prefix holds neither a volume nor an owner

Cause: The volume superblock and ownership marker are both gone from the bucket, most likely because the volume was wiped.

Fix: If the volume should not exist, detach, adding --allow-dirty-loss if it reports dirty slots. To recreate it from this cache, run update --device-size; only what the cache holds comes back, and the rest reads as zeros.

Exit code 9: the volume and this device disagree

Cause: The cache device’s records do not match the volume in the bucket that the grant names.

Fix: Confirm the grant is the right one. The message names what disagrees: location, volume, chunk or block size, or a leftover ublk device.

Ownership and volume identity

start, attach or detach refuses with exit code 13: owned elsewhere or superseded

Cause: The ownership marker in the bucket belongs to another cache, on another host or on this one.

Fix: Stop the daemon and detach where the marker points. To give this device up instead, detach it, adding --allow-dirty-loss if it was left unclean. If the owner is gone for good, attach --take-ownership --overwrite; read Ownership first for what that abandons.

start, update or detach refuses with exit code 13: the volume was released

Cause: Either a detach from this device released the marker but did not finish, or another host owned the volume after this cache last held it, so the cache is stale.

Fix: If the release was from this device, run detach again, adding --allow-dirty-loss if it was left unclean. Otherwise attach --overwrite, which lays out a fresh cache.

start, update or detach refuses with exit code 9: the cache belongs to a different volume

Cause: The volume at the prefix was created separately from the one this cache served, for example after a wipe and init.

Fix: Compare the volume id in flexblock show. If the new volume is the one you want, attach --overwrite, which discards this cache. To give the device up instead, detach --allow-dirty-loss.

init, attach and detach

init or attach refuses: the store does not enforce conditional writes (exit code 10)

Cause: The object store endpoint mishandled a conditional PUT (If-None-Match or If-Match), so it cannot keep two hosts from claiming the volume.

Fix: Use a store, or a version of it, that enforces conditional PUTs.

init, attach or start refuses on missing permissions (exit code 10)

Cause: The bucket policy or role lacks a permission on the prefix. FlexBlock checks by writing, reading, listing and deleting a scratch __access_probe/ object.

Fix: Grant read, write, list and delete on the bucket and prefix, then retry. See Credentials.

init refuses because the prefix is not empty (exit code 18)

Cause: The prefix holds keys, perhaps from an earlier volume or unrelated objects.

Fix: Use a different prefix. If the objects are a dead flexBlock volume, remove them with wipe --allow-unverified using that volume’s grant. Nothing overrides this check.

attach refuses because the device is unclean or holds another cache

Cause: The device was not cleanly stopped, or belongs to a different volume.

Fix: For an unclean device that still holds this volume (exit code 16), prefer start followed by a clean stop. A cache superseded since it last held the volume cannot be flushed; the refusal names the flags that discard it. Otherwise --overwrite, which discards a foreign cache.

detach refuses with dirty slots or pending discards (exit code 16)

Cause: The cache holds data or discards not yet applied to the bucket.

Fix: start the daemon, wait for flexblock_cache_dirty_slots and flexblock_discard_deletes_pending to reach 0, stop cleanly, then detach. --allow-dirty-loss discards the data instead; see Where Data Can Be Lost.

Growing a volume

update —device-size reports the device was only partly resized (exit code 4)

Cause: The volume’s superblock in the bucket was updated, but the local superblock rewrite or the live resize did not finish.

Fix: Follow the step the message names: re-run update --device-size, or restart the daemon.

update —device-size exits busy: the daemon did not answer on its control socket

Cause: The daemon has no listener (its start log says the control socket is not listening, or the socket under /run/flexblock was removed), or it did not reply within 180 seconds.

Fix: Stop the daemon, run update offline, then start and grow the filesystem.

update —device-size reports the kernel cannot resize a live device (exit code 10)

Cause: The kernel is older than 6.16 and lacks live resize support.

Fix: Grow offline: stop the daemon, run update, then start and grow the filesystem. See Growing a Volume.

Performance and errors

Writes to new regions are slow

Symptom: flexblock_writeback_throttle_wait_seconds_total climbs.

Cause: The dirty ratio passed --max-dirty-ratio, and write-back pressure is pacing writes that dirty new chunks to the upload rate.

Fix: Expected during a burst larger than the cache. If it persists with flexblock_obj_upload_capacity at the logged writeback boost ceiling and no flexblock_obj_retries_total, raise --concurrent-uploads or use a larger cache. See Write-Back Behavior.

Writes are slow and `flexblock_cache_write_stalls_total` climbs

Cause: Every slot holds dirty data and the flush loop cannot upload fast enough.

Fix: Check flexblock_obj_errors_total and flexblock_obj_retries_total for a bucket problem. Otherwise raise --concurrent-uploads, lower --target-dirty-ratio, or use a larger cache.

A filesystem reports read errors or remounts read-only on a healthy device

Cause: A read waited 15 seconds on a write that overlapped it, and failed with EIO rather than return a mix of two writes (flexblock_cache_read_stall_failures_total). The daemon log names the chunk.

Fix: Look for what stalled the write: flexblock_wal_head_stalls_total rising, and flexblock_flush_slices_total standing still.

Throughput drops sharply to a fixed ceiling

Cause: The license has expired past its grace period, or does not cover the volume, so guest I/O is throttled.

Fix: Check flexblock_license_state, or the license section of flexblock show <device>. Install the renewed grant; the daemon reloads it within a minute. See Licensing.

Nothing appears at the metrics port

Cause: Metrics are off by default.

Fix: Add --metrics-port to the start arguments. See Metrics Endpoint.