Stopping Cleanly

A clean stop is an unmount followed by a signal to the daemon. The daemon then uploads everything the cache still holds that object storage does not, and marks the cache device cleanly stopped.

Procedure

1. Unmount the filesystem (or tear down whatever else is using /dev/ublkbN).

   sudo umount /mnt/analytics

2. Send the daemon SIGINT, SIGTERM or SIGHUP.

   sudo kill -INT "$(pidof flexblock)"

   With more than one volume on the host, signal the PID that flexblock show reports instead of using pidof.

If the daemon runs under systemd, systemctl stop does both steps in the right order. See Running Under systemd.

What the daemon does

On the signal, the daemon:

1. Flushes all dirty chunks to object storage.
2. Finishes any pending discard deletes.
3. Checkpoints and marks the cache device cleanly stopped.

It has --shutdown-flush-timeout to do this, one hour by default. Anything still dirty or undeleted when the timeout runs out stays on the cache device for the next start to finish, and the stop exits with code 16 (not cleanly stopped). No data is lost in that case; it simply waits on NVMe until the daemon runs again.

Why the daemon refuses to stop while mounted

Stopping under a mounted filesystem would leave the filesystem blocked on a device with nothing behind it. So if the device is mounted, used for swap or claimed by another layer when the signal arrives, the daemon logs what is using it and keeps serving. Unmount or release the device, then send the signal again.

A process that merely holds the device open is not a reason to keep serving. It gets five seconds to close the device (or until a second signal). After that the stop goes ahead so the shutdown flush still runs, and that process sees I/O errors.

Do not use SIGKILL

A killed daemon cannot flush and leaves the cache unclean. The next start recovers, but dirty data waits on NVMe until then. Under systemd, keep TimeoutStopSec above --shutdown-flush-timeout so systemd never reaches for SIGKILL during a long flush.

Do not keep signaling during the flush either: once the stop is under way a second signal is only logged, and a third exits at once without flushing, leaving the cache unclean for the next start to recover.

Related

• Recovering From a Crash
• Running Under systemd
• Exit Codes