Metrics Reference

Every series the daemon exposes, grouped by subsystem. All names carry the flexblock_ prefix, which the tables below omit: cache_hits_total is flexblock_cache_hits_total in full. Histograms emit the usual _bucket, _sum and _count series in seconds, with 19 upper bounds from 5 microseconds to 5 seconds plus +Inf (see Metrics Endpoint).

Tip

Start with the dozen series on the Alerting page. Most of the series below are for diagnosing a problem once an alert has fired.

Labeled series

Most series carry no labels. These do:

Series	Labels
bridge_read_fallbacks_total	reason: chunk_crossing, miss, pages_invalid, raced, writer_active
bridge_write_fallbacks_total	reason: clean_hit, exclusive, miss, page_padding, read_pressure
build_info	version
cache_policy_info	policy: lru or twoq-meta
license_info	license_id, customer, key_id, state: valid, grace, expired, uncovered, pending (unknown before the first grant)

Guest I/O (ublk)

What the filesystem or application on /dev/ublkbN sees.

Series	Type	Description
ublk_bytes_discarded_total	counter	Bytes covered by guest DISCARD requests
ublk_bytes_read_total	counter	Bytes returned to the guest by successful reads
ublk_bytes_written_total	counter	Bytes accepted from the guest by successful writes
ublk_bytes_zeroed_total	counter	Bytes covered by guest WRITE_ZEROES requests
ublk_discard_duration_seconds	histogram	Guest-visible DISCARD latency
ublk_discards_failed_total	counter	Guest DISCARD requests completed with an error
ublk_discards_total	counter	Guest DISCARD requests received
ublk_flush_duration_seconds	histogram	Guest-visible FLUSH latency, including the cache device sync
ublk_flushes_failed_total	counter	Guest FLUSH requests completed with an error
ublk_flushes_total	counter	Guest FLUSH requests received; not the same as flush_attempts_total, which counts uploads to object storage
ublk_fua_writes_total	counter	Guest writes carrying FUA; each is durable before it is acknowledged
ublk_metadata_ops_total	counter	Guest reads and writes the kernel tagged as filesystem metadata; under twoq-meta their chunks go straight to the protected Am tier
ublk_ops_failed_total	counter	Guest requests of every kind completed with an error
ublk_ops_unsupported_total	counter	Guest requests rejected as unsupported; non-zero means the kernel issued an operation the device does not advertise
ublk_read_duration_seconds	histogram	Guest-visible READ latency, including any license throttling
ublk_reads_failed_total	counter	Guest READ requests completed with an error
ublk_reads_total	counter	Guest READ requests received
ublk_write_duration_seconds	histogram	Guest-visible WRITE latency, including throttling and FUA syncs
ublk_write_zeroes_duration_seconds	histogram	Guest-visible WRITE_ZEROES latency
ublk_write_zeroes_failed_total	counter	Guest WRITE_ZEROES requests that failed
ublk_write_zeroes_total	counter	Guest WRITE_ZEROES requests received
ublk_writes_failed_total	counter	Guest WRITE requests that failed, FUA sync failures included
ublk_writes_total	counter	Guest WRITE requests received

Bridge

Most guest I/O is served on a fast path straight from the cache device. Requests the fast path cannot serve, such as cache misses, are handed to a slower path (the “bridge”). These series show how much I/O takes that path and why.

Series	Type	Description
bridge_capacity	gauge	Maximum guest requests the slow path serves at once: queues times queue depth
bridge_inflight	gauge	Guest requests currently on the slow path
bridge_read_fallbacks_total	counter	Guest reads sent to the slow path, by reason: the request spans chunks, the chunk is not cached, its pages are not valid, a writer was active, or the read raced a writer
bridge_write_fallbacks_total	counter	Guest writes sent to the slow path, by reason: the chunk is cached but clean, a merge or discard needs it exclusively, it is not cached, page padding is needed, or a waiting reader overlaps the write

Cache

Hits, misses, occupancy and the cache replacement policy. The am_* and a1out_* series stay zero under the lru policy by design; see Cache Replacement Policy.

Series	Type	Description
cache_a1in_admissions_total	counter	Chunks admitted to the probationary A1in tier; under lru this is every admission
cache_a1in_evictions_total	counter	Chunks evicted from A1in to free a slot; under lru this is every eviction
cache_a1in_removals_total	counter	A1in entries dropped by discard, write rollback or recovery rather than eviction
cache_a1in_restored_total	counter	Entries placed into A1in by crash recovery
cache_a1in_size	gauge	Entries resident in A1in, dirty included
cache_a1out_expirations_total	counter	Ghost identities in A1out that aged out without being reused
cache_a1out_forgotten_total	counter	A1out ghosts dropped by discard, removal or recovery
cache_a1out_insertions_total	counter	Chunk identities recorded in A1out on eviction from A1in
cache_a1out_meta_hits_total	counter	A1out ghosts consumed by a metadata admission rather than plain reuse
cache_a1out_promotions_total	counter	Chunks admitted straight to Am because they were reused while in A1out
cache_a1out_size	gauge	Chunk identities in the A1out ghost list (identities only, no data)
cache_am_direct_admissions_total	counter	Chunks admitted directly to Am by a filesystem metadata I/O
cache_am_evictions_total	counter	Chunks evicted from the protected Am tier to free a slot
cache_am_meta_promotions_total	counter	A1in chunks promoted to Am at eviction time because they hold filesystem metadata
cache_am_removals_total	counter	Am entries dropped by discard, write rollback or recovery rather than eviction
cache_am_restored_total	counter	Entries placed into Am by crash recovery
cache_am_size	gauge	Entries resident in Am, dirty included
cache_dirty_slots	gauge	Cache slots holding data not yet uploaded; divide by cache_total_slots for the ratio --max-dirty-ratio and --target-dirty-ratio act on
cache_eviction_waits_total	counter	Times a cache allocation waited because every eviction candidate was in use
cache_hits_total	counter	Reads served from the cache device
cache_lock_read_wait_seconds	histogram	Wait for contended shared access to the cache map; rising millisecond waits mean flushes or discards are holding up guest reads
cache_lock_write_hold_seconds	histogram	Time the cache map was held exclusively; normally microseconds, and a long tail blocks concurrent readers
cache_lock_write_wait_seconds	histogram	Wait for contended exclusive access to the cache map; scales with --concurrent-uploads
cache_metadata_bit_set_total	counter	Chunks marked as holding filesystem metadata
cache_metadata_entries	gauge	Resident chunks marked as holding filesystem metadata
cache_misses_total	counter	Reads that had to fetch the chunk from object storage
cache_occupied_slots	gauge	Cache slots holding a chunk, clean or dirty
cache_oldest_dirty_age_seconds	gauge	Age of the oldest dirty chunk: how far the cache lags object storage, and how much acknowledged data a cache device failure would cost; 0 when nothing is dirty
cache_policy_info	gauge	Active cache replacement policy, in the policy label
cache_re_dirty_total	counter	Writes that landed on a chunk a concurrent flush had just cleaned, making it dirty again
cache_read_duration_seconds	histogram	Time per cache read
cache_read_stall_failures_total	counter	Guest reads failed with EIO because a write they waited on did not finish in time; a torn read is refused rather than returned
cache_second_chances_total	counter	Evictions declined because the chunk was used again recently; under lru this is the whole recency mechanism
cache_total_slots	gauge	Cache slots laid out on the cache device
cache_unpin_miscount_total	counter	Internal consistency violations; any non-zero value is a bug to report to Paradigm4
cache_write_duration_seconds	histogram	Time per cache write
cache_write_stalls_total	counter	Times a write waited for a cache slot because every slot held dirty data
prefetch_completed_total	counter	Chunks fetched ahead of a sequential cold read
prefetch_failed_total	counter	Prefetches that failed to fetch or cache their chunk
prefetch_issued_total	counter	Chunk prefetches started by sequential cold reads
rmw_merges_total	counter	Chunks whose missing pages were fetched from object storage and merged because a read, partial write or zeroing needed them

Flush and write-back

Uploads of dirty chunks to object storage.

Series	Type	Description
flush_attempts_total	counter	Chunk uploads issued, failures included
flush_claim_skips_total	counter	Chunks a flush pass skipped because another pass already had them
flush_duration_seconds	histogram	Time per chunk flush attempt, waits and retries included
flush_errors_total	counter	Flush tasks that failed at any stage
flush_eviction_pressure_total	counter	Dirty chunks flushed early because a writer needed a slot and found nothing evictable
flush_exhausted	gauge	1 after a flush batch exhausted its retry budget with no successful upload since; while set, a write that finds the cache full fails instead of waiting
flush_loop_iterations_total	counter	Passes of the flush loop; if this and flush_slices_total are flat while cache_oldest_dirty_age_seconds climbs, the loop is stuck (unless the ownership lease is lapsed or a data sync failed)
flush_noops_total	counter	Flushes that found nothing to upload
flush_overtaken_total	counter	Uploads overtaken by a new write; the chunk stays dirty only for the newer writes
flush_quiet_skips_total	counter	Times the age-based pass skipped a chunk written within --flush-quiet
flush_skips_total	counter	Flushes that stood down because a write was in flight on the chunk; steady growth means a chunk cannot flush
flush_slices_total	counter	Batches of chunks flushed; keeps climbing while a long pass drains a backlog
flush_wait_for_blocker_permit_duration_seconds	histogram	Wait for a reserved upload slot by an urgent flush that writes are waiting on; compare with flush_wait_for_permit_duration_seconds
flush_wait_for_permit_duration_seconds	histogram	Wait for an upload slot
flush_wait_for_read_permit_duration_seconds	histogram	Wait for a slot to read the chunk from the cache device
flush_write_holds_total	counter	Flushes of an overdue chunk that briefly held off its writes, so a constantly written chunk still flushes

Object storage

Series	Type	Description
obj_blocker_upload_capacity	gauge	Upload slots reserved for urgent flushes, on top of obj_upload_capacity
obj_blocker_upload_inflight	gauge	Reserved urgent upload slots in use
obj_bytes_downloaded_total	counter	Compressed bytes fetched from object storage
obj_bytes_uploaded_total	counter	Compressed bytes uploaded to object storage by flushes
obj_cleanup_inflight	gauge	Background object-storage tasks in flight: discard deletes and their retries, and read prefetch
obj_delete_batches_total	counter	Delete batches issued; each request holds up to 1000 keys (256 on Blob, 100 on GCS)
obj_delete_capacity	gauge	Maximum discard operations in flight
obj_delete_discard_batches_inflight	gauge	Discard-driven delete batches in flight or waiting for delete budget
obj_delete_inflight	gauge	Discard operations currently holding a delete slot
obj_delete_keys_per_second	gauge	Delete pace on AWS S3 in keys per second; 0 when unpaced
obj_deletes_total	counter	Objects deleted by discards
obj_download_capacity	gauge	Maximum concurrent GETs for misses and prefetch (--concurrent-downloads)
obj_download_inflight	gauge	GETs currently in flight for misses and prefetch
obj_downloads_total	counter	Successful chunk fetches for any purpose
obj_errors_total	counter	Operations that failed permanently or exhausted their retries (--retry-cutoff)
obj_flush_fetch_capacity	gauge	Maximum concurrent GETs by flushes filling in missing pages
obj_flush_fetch_inflight	gauge	Flush GETs in flight
obj_get_duration_seconds	histogram	Time per chunk fetch, retries included
obj_put_duration_seconds	histogram	Time per chunk upload, retries included
obj_retries_total	counter	Retries after a transient error
obj_upload_capacity	gauge	Current maximum concurrent PUTs: --concurrent-uploads, raised up to four times that under write-back pressure
obj_upload_inflight	gauge	Uploads in flight
obj_uploads_total	counter	Successful chunk uploads

Object index and discards

Series	Type	Description
delete_queue_discard	gauge	Discarded chunk keys queued for deletion
discard_deletes_pending	gauge	Discarded chunks whose object delete has not completed; must reach 0 before detach
discard_redrive_pending	gauge	Failed discard deletes awaiting retry
object_index_entries	gauge	Chunks that have an object in the bucket; this index grows in memory as the volume fills

NVMe cache device

Series	Type	Description
device_syncs_total	counter	Cache device syncs for guest FLUSH and FUA requests; stays 0 when the device has no volatile write cache and uses direct I/O
meta_bitmap_persist_failures_total	counter	Failed passes writing slot page state to the cache device; unwritten state is lost on a crash
meta_bitmap_persist_pending	gauge	Chunks whose page state changed in memory and is not yet on the cache device
meta_bitmap_persist_stale_skips_total	counter	Queued page-state records dropped because the chunk changed before they were written
meta_page_writes_total	counter	4 KiB slot metadata pages written
meta_record_updates_total	counter	Slot metadata record updates; divide by meta_page_writes_total for the batching factor
meta_write_errors_total	counter	Slot metadata page writes that failed
nvme_buffered_reads_total	counter	Chunk reads through the page cache; in direct mode, reads that O_DIRECT could not express
nvme_circuit_breaker_open	gauge	1 while consecutive cache device errors have tripped the circuit breaker and new fast-path I/O is rejected
nvme_dio_stage_total	counter	Chunk I/Os copied through an aligned staging buffer to stay on O_DIRECT
nvme_dio_unaligned_total	counter	Chunk I/Os that fell back to buffered I/O because they were not 4096-aligned
nvme_direct_reads_total	counter	Chunk reads through O_DIRECT
nvme_errors_total	counter	I/O errors on the cache device; 64 in a row open the circuit breaker
nvme_fdatasync_fallback_total	counter	Durable writes done as a write plus fdatasync, because RWF_DSYNC is unsupported or --no-fua is set
nvme_read_duration_seconds	histogram	Time per cache device chunk read
nvme_write_duration_seconds	histogram	Time per cache device write (data, metadata, log and superblocks)

Write-ahead log

The write-ahead log (WAL) is a ring of intents on the cache device that crash recovery replays. These series also cover write-back pacing.

Series	Type	Description
wal_active_entries	gauge	Dirty intents whose flush has not landed; add discard_deletes_pending and divide by wal_ring_capacity for ring utilization
wal_checkpoint_failures_total	counter	Log checkpoints that failed to persist; recovery scans from the last one that did
wal_head_backstop_flushes_total	counter	Urgent ring-head chunks the main flush loop flushed alongside the dedicated worker
wal_head_flush_batches_total	counter	Batches of ring-head chunks flushed by the dedicated urgent worker
wal_head_flush_duration_seconds	histogram	Time the urgent worker spent flushing a batch; what a stalled write waits on
wal_head_rotations_total	counter	Active entries moved past the ring head; steady growth means write-back is more than a lap behind, a chunk cannot flush, or a large trim is slow
wal_head_stalls_total	counter	Writes and discards that waited because the whole ring held active entries
wal_mark_completed_failures_total	counter	Failures to persist an entry’s completion; the entry is reclaimed later
wal_orphans_recovered_total	counter	Orphaned entries reclaimed at the ring head; steady growth deserves a look
wal_records_since_checkpoint	gauge	Records written since the last checkpoint
wal_ring_capacity	gauge	Total ring entries
writeback_pressure	gauge	Write-back pressure from 0 to 1: 0 at or below --max-dirty-ratio, 1 halfway from there to a full cache
writeback_throttle_chunks_per_second	gauge	Rate at which writes may dirty new chunks while write pacing is on; 0 when off
writeback_throttle_wait_seconds_total	counter	Seconds guest writes spent delayed by write pacing

Compression

Series	Type	Description
compress_bytes_in_total	counter	Uncompressed bytes handed to the compressor by flushes
compress_bytes_out_total	counter	Compressed bytes produced, headers included; in over out is the compression ratio
compress_duration_seconds	histogram	Time per chunk compression
decompress_duration_seconds	histogram	Time per chunk decompression after a fetch

License enforcement

Series	Type	Description
license_expires_at_seconds	gauge	Unix time the grant in force expires
license_grace_ends_at_seconds	gauge	Unix time the grace period ends and throttling engages
license_info	gauge	Identity and state of the grant in force, in labels
license_reload_failed	gauge	1 if the last poll could not re-read the grant file and the last good grant is still in use
license_reload_failures_total	counter	Polls that could not re-read the grant file
license_state	gauge	0 valid, 1 grace, 2 expired, 3 not covered by the grant, 4 not yet in force; 2 and above throttle guest I/O
license_throttle_read_bytes_per_second	gauge	Read bandwidth limit being applied; 0 when unthrottled
license_throttle_wait_seconds_total	counter	Seconds guest I/O spent delayed by license throttling
license_throttle_waits_total	counter	Guest reads and writes delayed by license throttling
license_throttle_write_bytes_per_second	gauge	Write bandwidth limit being applied; 0 when unthrottled

Ownership

Series	Type	Description
ownership_check_failures_total	counter	Ownership re-checks that could not complete; serving continues, but bucket writes pause once the lease lapses
ownership_lease_held	gauge	1 while the ownership lease is held; 0 while bucket writes are paused
ownership_marker_reestablished_total	counter	Times the ownership marker was missing from the bucket and written back

Process, build and volume

Series	Type	Description
build_info	gauge	FlexBlock build serving this device, in the version label
chunk_size_bytes	gauge	Chunk size of the volume
device_size_bytes	gauge	Size of the block device presented to the guest
jemalloc_active_bytes	gauge	Bytes in active memory pages, per the allocator
jemalloc_allocated_bytes	gauge	Bytes the daemon has allocated, per the allocator
jemalloc_mapped_bytes	gauge	Bytes in mapped memory, per the allocator
jemalloc_resident_bytes	gauge	Bytes in physically resident pages, per the allocator
logical_block_size_bytes	gauge	Logical block size the device advertises
start_time_seconds	gauge	Unix time this daemon started

Related

• Metrics Endpoint
• Alerting
• Common Issues