Ownership

A volume can be served by any host with its grant and a cache device, but only one host at a time. FlexBlock enforces that with an ownership marker in the bucket and a write lease held by the serving daemon. Two hosts writing the same volume at once would corrupt it; ownership is what prevents that.

The ownership marker

The marker is the __owner object under the volume’s prefix. It records the owner id, hostname, cache device, claim time, and whether the volume was taken over or released. Apart from this marker, nothing about the serving host is written into the bucket.

attach writes the marker and mirrors it into the cache device.
detach does not delete it, but rewrites it as released, so the bucket remembers that the volume has changed hands.
Every change to the marker is a conditional write (create-if-absent or compare-and-swap), so two hosts cannot both claim the volume.

Checks before claiming

init, attach and start first check, with scratch probe objects, that the credentials can write, overwrite, read, list and delete. init and attach also check that the store enforces conditional writes. They refuse a store that fails either check, because without conditional writes ownership cannot be enforced.

The write lease

While serving, the daemon reasserts the marker every 15 seconds, and each confirmation renews a 45-second lease.

While the lease is held, the daemon writes to the bucket normally.
If the marker cannot be confirmed, for example because the bucket is unreachable, the lease lapses. The daemon keeps serving I/O from the cache but starts no bucket writes until the lease is renewed.
Any single bucket write still running 240 seconds after it began is abandoned.

The flexblock_ownership_lease_held and flexblock_ownership_check_failures_total metrics show the lease’s health; see Alerting.

Taking ownership

The normal way to move a volume is a clean stop and detach on the old host, then attach on the new one. See Moving a Volume.

When the old host is gone and cannot detach, the new host claims the volume with attach --take-ownership, which replaces the marker with a compare-and-swap:

sudo flexblock attach /dev/nvme0n1p2 --take-ownership

If the old daemon is in fact still running, it starts no new bucket writes once its 45-second lease lapses, abandons any write still in flight after 240 seconds, and at its next check stops without flushing and exits with the ownership exit code (13), leaving its cache unclean.

Why the first start waits 285 seconds

After a take-over, the new owner’s first start waits 285 seconds before it serves or writes to the bucket. That is the old owner’s 45-second lease plus the 240-second limit on any write it might still have in flight. By the time the wait ends, the old daemon can no longer write, so nothing it sent can land on top of the new owner’s data.

The same wait applies in two other cases, since the daemon that used the old cache may still be running elsewhere:

detach of a cache that was stopped uncleanly.
attach --overwrite laying out a fresh cache over an existing one.

The fence assumes both hosts’ clocks run at the same rate and that an abandoned write never lands later. The lease is timed on a clock that keeps counting while the host is suspended, but a daemon frozen between confirming its lease and sending a write, for longer than the fence, can still send that write when it resumes.

Stale caches are refused

A cache is served only while nobody else has owned its volume since it was last in step with it:

start and update refuse a cache whose volume another host has owned since, or that was re-created at the same prefix.
start refuses when neither the marker nor the volume superblock exists, since the volume may have been wiped.
attach never keeps a superseded cache; laying out a fresh one requires --overwrite.

These refusals exit with code 13 (owned elsewhere or superseded) or 9 (the volume and this device disagree). The message names the state it found and the flag that would override it. See Common Issues.