Alerting

The daemon exposes more than a hundred series, but a small set covers the failure modes that need a person: write-back falling behind, object storage or the cache device failing, guest reads failing, and the license about to throttle the volume. This page lists them and gives example Prometheus rules.

Enable the endpoint first; see Metrics Endpoint.

What to alert on

Series names in the table below omit the flexblock_ prefix: flush_exhausted is flexblock_flush_exhausted in full. The rules further down use full names, as Prometheus sees them.

Series	Watch for	What it means
`cache_oldest_dirty_age_seconds`	Sustained above twice `--flush-threshold`	Write-back lag. The flush loop is not keeping up with the write rate, or uploads are failing. This is also how much acknowledged data a cache device failure would cost right now, though what remains in the bucket may need a filesystem check.
`flush_exhausted`	Equal to 1	A flush batch exhausted its retry budget with no successful upload since. While set, a write that finds the cache full fails instead of waiting.
`obj_errors_total`	Increasing	Object-storage operations failed permanently or exhausted their retries. Pair with `obj_retries_total` to tell a blip from an outage.
`nvme_circuit_breaker_open`	Equal to 1	64 consecutive cache device errors tripped the breaker; new cached I/O is rejected rather than served from a possibly corrupt device.
`cache_read_stall_failures_total`	Increasing	A guest read failed with EIO behind a stalled write. Filesystems treat this as corruption.
`nvme_errors_total`	Increasing	I/O errors on the cache device. Steady growth warrants checking the device’s health.
`wal_active_entries`	Approaching `wal_ring_capacity`	Write-ahead log ring pressure. Add `discard_deletes_pending`, whose intents hold ring entries too. Writes wait only when every entry is active.
`cache_dirty_slots`	Near `--max-dirty-ratio`	As a share of `cache_total_slots`: the cache is filling with unflushed data, and writes that dirty new chunks start to be paced.
`writeback_throttle_chunks_per_second`	Above 0 for minutes	Writes that dirty new chunks are being paced to the upload rate.
`license_state`	2 or above	Expired, uncovered or pending: guest I/O is throttled. 1 is grace; warn on it.
`license_expires_at_seconds`	Less than 14 days away	Time to arrange a renewal.
`ownership_check_failures_total`	Increasing	Ownership cannot be confirmed. Serving continues, but bucket writes pause, and `ownership_lease_held` reads 0, once the 45-second lease lapses. The bucket may be down.

With the defaults (--flush-threshold of 4 minutes), “twice --flush-threshold” is 8 minutes (480 seconds), which the example rule below uses. If you change the threshold, change your alert with it. See Write-Back Behavior.

Example rules

These rules come from the flexBlock User Guide. Put them in a rules file referenced from prometheus.yml and add labels and annotations to suit your alert routing.

groups:
  - name: flexblock
    rules:
      - alert: FlexBlockWritebackLagging
        # 480 s is twice the default --flush-threshold; scale it if you change the threshold.
        expr: flexblock_cache_oldest_dirty_age_seconds > 480
        for: 10m
      - alert: FlexBlockFlushExhausted
        expr: flexblock_flush_exhausted == 1
      - alert: FlexBlockCacheDeviceBreakerOpen
        expr: flexblock_nvme_circuit_breaker_open == 1
      - alert: FlexBlockLicenseThrottled
        expr: flexblock_license_state >= 2
      - alert: FlexBlockLicenseExpiringSoon
        expr: flexblock_license_expires_at_seconds - time() < 14 * 86400

Expressions for the remaining signals

The guidance for the other signals is qualitative (“increasing”, “approaching”, “for minutes”), so the time windows, ratios and durations are yours to choose. Replace each <window>, <ratio> and <duration> placeholder with a value that suits your environment.

groups:
  - name: flexblock-extra
    rules:
      # Pick the window; compare with flexblock_obj_retries_total to tell a blip from an outage.
      - alert: FlexBlockObjectStorageErrors
        expr: increase(flexblock_obj_errors_total[<window>]) > 0
      # Any increase means a guest read failed with EIO.
      - alert: FlexBlockReadStallFailures
        expr: increase(flexblock_cache_read_stall_failures_total[<window>]) > 0
      # Pick the window, and whether one error pages or only a steady rate.
      - alert: FlexBlockCacheDeviceErrors
        expr: increase(flexblock_nvme_errors_total[<window>]) > 0
      # Pick the ratio at which you want warning.
      - alert: FlexBlockWalRingPressure
        expr: (flexblock_wal_active_entries + flexblock_discard_deletes_pending) / flexblock_wal_ring_capacity > <ratio>
      # Pick a ratio just below --max-dirty-ratio (0.8 by default).
      - alert: FlexBlockDirtyCacheShare
        expr: flexblock_cache_dirty_slots / flexblock_cache_total_slots > <ratio>
      # Pick a duration of some minutes, so short bursts do not page.
      - alert: FlexBlockWritePacing
        expr: flexblock_writeback_throttle_chunks_per_second > 0
        for: <duration>
      # Route as a warning, not a page.
      - alert: FlexBlockLicenseInGrace
        expr: flexblock_license_state == 1
      # Pick the window; flexblock_ownership_lease_held == 0 means bucket writes are paused.
      - alert: FlexBlockOwnershipChecksFailing
        expr: increase(flexblock_ownership_check_failures_total[<window>]) > 0

Capacity signals

These are not alerts, but they are worth graphing:

Hit rate: flexblock_cache_hits_total against flexblock_cache_misses_total.
Stored chunks: flexblock_object_index_entries, the chunks that have an object in the bucket.
Compression ratio: flexblock_compress_bytes_in_total over flexblock_compress_bytes_out_total.