Running Under systemd

In production the flexBlock daemon runs as a systemd service, with the filesystem mount or the NVMe-oF export ordered after it. The units on this page and the pages that follow encode the two ordering rules that matter:

• Nothing uses the device before it is live.
• Everything lets go of the device before the daemon is signaled.

The examples assume the binary lives in /usr/local/bin, the cache device is /dev/nvme1n1p1, and flexblock attach has already been run once by hand.

The readiness gate

flexblock start runs in the foreground and has no readiness protocol of its own. A unit that simply launched it would report “started” before /dev/ublkbN exists. The template closes that gap with an ExecStartPost that polls flexblock show --json until the daemon has recorded a live device. Two things follow:

• A unit with Requires= and After= on the flexBlock service starts only once the block device is serving.
• On the way down, systemd stops that unit first, because After= ordering is reversed at shutdown.

The second point matters. The daemon refuses a signal-driven stop while its device is mounted, used for swap or claimed, but it cannot see an NVMe-oF export, which the kernel holds open without claiming it. An export left in place would start failing I/O. Consumers must be torn down before the daemon is signaled, and the dependency ordering does that for you.

The service unit

Install this as /etc/systemd/system/flexblock@.service. It is a template whose instance name is the cache device under /dev, so flexblock@nvme1n1p1.service serves /dev/nvme1n1p1.

[Unit]
Description=flexBlock volume on /dev/%i
Wants=network-online.target
After=network-online.target dev-%i.device
Requires=dev-%i.device

[Service]
Type=exec
EnvironmentFile=-/etc/flexblock/%i.env
ExecStartPre=/sbin/modprobe ublk_drv
ExecStart=/usr/local/bin/flexblock start /dev/%i $FLEXBLOCK_START_ARGS
ExecStartPost=/bin/sh -c "until /usr/local/bin/flexblock show /dev/%i --json 2>/dev/null | grep -q '\"pid_alive\": true'; do kill -0 $MAINPID || exit 1; sleep 1; done"
TimeoutStartSec=30min
TimeoutStopSec=75min
Restart=on-failure
RestartSec=10
RestartMode=direct
RestartPreventExitStatus=2 7 8 9 10 11 12 13 15 17
LimitMEMLOCK=infinity
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

What the less obvious settings do:

Setting	Why
ExecStartPre=/sbin/modprobe ublk_drv	Loads the ublk driver before each start.
ExecStartPost=...	The readiness gate. It gives up if the daemon dies first.
TimeoutStartSec=30min	Rebuilding the object index at start scales with the number of chunks in the bucket.
TimeoutStopSec=75min	SIGTERM triggers the same graceful stop as Ctrl+C. This allows the full shutdown flush before systemd sends SIGKILL.
RestartMode=direct	Restarts the daemon in place after a crash. See Restart behavior.
RestartPreventExitStatus=...	Does not loop on failures a restart cannot fix: usage, newer build, declined, mismatch, preflight, not initialized, busy, owned elsewhere, unsafe, license. See Exit Codes.

The environment file

Per-instance options go in an environment file named after the device, here /etc/flexblock/nvme1n1p1.env:

FLEXBLOCK_START_ARGS=--ublk-dev-id 0 --metrics-port 10056

Pin the ublk id with --ublk-dev-id so the device path (/dev/ublkb0 here) stays the same across restarts. The mount and export units depend on it.

Credentials can live in this file too when an instance role is not available. In that case, keep the file readable by root only.

Enable and start

1. Reload systemd and start the service.

   sudo systemctl daemon-reload
   sudo systemctl enable --now flexblock@nvme1n1p1.service

2. Add a consumer: an XFS or ext4 mount, or an NVMe-oF/TCP Export.

Stop timeouts

Keep TimeoutStopSec above the shutdown flush timeout

If systemd reaches its stop timeout first, it sends SIGKILL, which leaves the cache unclean. The next start recovers, but any dirty data waits on NVMe until then. The template’s 75 minutes leaves room above the default --shutdown-flush-timeout of one hour; raise both together if you change it.

Restart behavior

If the daemon crashes, or exits with code 16 after a queue thread fails (a stop that runs out of --shutdown-flush-timeout with data left also exits 16), Restart=on-failure starts it again after ten seconds and start recovers the quiesced ublk device. A mounted filesystem blocks for at least that long, then carries on.

RestartMode=direct, which needs systemd 254 or newer, is what makes this work. In the default mode, systemd propagates the restart to every unit that Requires= the service and tries to unmount the filesystem first. That unmount blocks on the quiesced device until the daemon is back, which it never is, because the start waits for the unmount.

Note

The unit deliberately sets no RuntimeDirectory=. The daemon creates /run/flexblock itself. systemd would remove that directory whenever the service stops, including before a restart after a crash, taking the state the restarted daemon needs to recover writes not yet flushed.

Device timeout drop-in

Mount units gain an implicit dependency on the device unit behind What=, and systemd waits only 90 seconds by default for a device to appear. A cold start can take longer than that, so raise the limit for the pinned ublk device with a drop-in at /etc/systemd/system/dev-ublkb0.device.d/timeout.conf:

[Unit]
JobRunningTimeoutSec=30min

The fstab examples on the filesystem pages carry the same setting as x-systemd.device-timeout=30min.

If systemctl stop hangs

If systemctl stop of the flexBlock service hangs, the daemon almost certainly received the signal while the device was still in use (mounted, perhaps in another mount namespace, or claimed by another layer) and refused it. The journal shows device is still in use.

1. Unmount or release the device by hand.

2. Send the signal again.

   sudo systemctl kill -s SIGTERM flexblock@nvme1n1p1.service

Otherwise systemd waits for TimeoutStopSec and then sends SIGKILL. Once the flush is under way, do not signal again: a third signal exits without flushing.

Next steps

• XFS Filesystem
• ext4 Filesystem
• NVMe-oF/TCP Export
• Stopping Cleanly