Performance

These results compare flexBlock with the NVMe it runs on. One instance, one drive: a raw partition of the local NVMe against the flexBlock device whose cache is a sibling partition of the same drive, plus cold reads from the bucket and PostgreSQL and MariaDB benchmarks on both.

Each fio number is one 30-second run after a 5-second ramp, except the cold sequential reads, which cover the region once. Each database number is a full pgbench or sysbench run.

Headline	Result
Sequential throughput	100% of raw NVMe
Warm random 4 KiB read IOPS	605k
Cold sequential read from S3, prefetch 64	2.6 GiB/s
MariaDB write-only OLTP	117% of raw NVMe

Raw NVMe versus flexBlock, warm

fio over io_uring with O_DIRECT on a 20 GiB region that is resident in the cache and already flushed: the steady state of a working set that fits.

Throughput, MiB/s

Test	Raw NVMe	flexBlock	Ratio
Sequential read, 1 MiB, QD32	4,069	4,069	100%
Sequential write, 1 MiB, QD32	3,212	3,213	100%
Random read, 128 KiB, QD16	4,117	4,069	99%

Random 4 KiB, IOPS

Test	Raw NVMe	flexBlock	Ratio
Read, QD1 (74 vs 83 µs)	13,331	11,805	89%
Read, QD32, one job	336,876	242,017	72%
Read, QD32, four jobs	584,602	604,909	103%
Write, QD1 (16 vs 19 µs)	56,929	48,074	84%
Write, QD32, one job	330,675	298,443	90%
Write, QD32, four jobs	330,678	330,035	100%
O_SYNC write, QD1 (17 vs 23 µs)	55,646	41,074	74%

Mixed 70/30 read/write, 16 KiB, QD16, four jobs

Test	Raw NVMe	flexBlock	Ratio
Total IOPS	219,633	182,212	83%

Database: PostgreSQL pgbench

PostgreSQL 18 on XFS, scale 500, 32 clients, an fsync on every commit. TPC-B mean latency was 0.58 ms raw and 0.69 ms on flexBlock.

Workload	Raw NVMe	flexBlock	Ratio
Load 50 M accounts (scale 500)	20.4 s	23.1 s	88%
TPC-B read/write, 120 s, tps	55,276	46,045	83%
Select-only, 60 s, tps	424,940	417,717	98%

Database: MariaDB with sysbench

MariaDB 11.8 on XFS, sysbench OLTP, 16 tables of 1 M rows, 32 threads, a redo flush on every commit. Read/write p95 latency was 3.89 ms raw and 3.49 ms on flexBlock.

Workload	Raw NVMe	flexBlock	Ratio
Load 16 M rows	13.8 s	13.6 s	101%
OLTP read/write, 120 s, tps	13,648	14,077	103%
OLTP read-only, 60 s, tps	20,788	19,963	96%
OLTP write-only, 60 s, tps	49,542	57,912	117%

Cold reads from object storage

After detach and attach the cache is empty, so every read is a miss served from the bucket. Sequential reads trigger the read prefetch. A random 4 KiB miss fetches its whole chunk, so the region warms within the run and later reads hit NVMe.

Cold read	Result
Seq 1 MiB QD32, 20 GiB, prefetch 64	2,637 MiB/s
Same read, default prefetch 32	1,449 MiB/s
Mean / p99 latency, prefetch 64	12 / 39 ms
Mean / p99 latency, default	22 / 76 ms
Chunks fetched, bytes moved	10,240, 17.5 GiB
Random 4 KiB QD32, 30 s, cold	157,717 IOPS
Region fully resident after	under 30 s

The 2.6 GiB/s cold figure uses --read-prefetch 64 and --concurrent-downloads 128. The defaults, which favor mixed workloads because deep prefetch competes with foreground I/O, gave 1.4 GiB/s on the same region in the same session.

Compression helps a cold read as far as the data compresses. This run’s random buffers barely compressed, so 17.5 GiB crossed the network for 20 GiB read, while an earlier run whose data compressed two to one moved 9.6 GiB.

What the numbers mean

With the working set resident in the cache, flexBlock performs like the NVMe drive itself:

Sequential throughput and multi-queue random reads and writes match the raw partition within noise.
The mixed read/write run reaches 83% of raw.
The read-only database runs stay within a few percent of raw.
MariaDB write-only beats raw by 17%, and pgbench’s load and TPC-B reach 88% and 83%.

Every write is acknowledged from NVMe and every warm read is served from it, so bandwidth-bound and multi-queue workloads see the drive, not the daemon. Each request does make a short round trip through the kernel to the flexBlock daemon, a few microseconds. That cost shows only at queue depth one or on a single submitting thread: about 9 µs on a lone 4 KiB read, 72% of raw for one thread at queue depth 32, and 74% for O_SYNC writes. More queues close the gap.

Cold reads are bounded by the object store and the prefetch depth, not by the cache device.

Latency

Latency in microseconds, mean / p99, from the same runs. Queue-depth-32 latencies are dominated by queueing, so compare like with like: the QD1 rows are the per-request cost, and the mixed rows show tail latency while the flush loop uploads in the background.

Test	Raw NVMe	flexBlock
Sequential read, 1 MiB, QD32	7,835 / 10,027	7,843 / 17,170
Sequential write, 1 MiB, QD32	9,918 / 13,697	9,927 / 22,151
Random read, 4 KiB, QD1	74 / 193	83 / 205
Random read, 4 KiB, QD32	94 / 138	131 / 224
Random read, 4 KiB, QD32 ×4	218 / 415	210 / 280
Random write, 4 KiB, QD1	16 / 32	19 / 26
Random write, 4 KiB, QD32	95 / 224	106 / 222
Random write, 4 KiB, QD32 ×4	386 / 610	386 / 485
Mixed 70/30, 16 KiB, reads	320 / 618	393 / 2,768
Mixed 70/30, 16 KiB, writes	217 / 415	246 / 2,342
Random read, 128 KiB, QD16	482 / 692	487 / 593
O_SYNC write, 4 KiB, QD1	17 / 31	23 / 99

Tail latency in the mixed run is where write-back is visible: while dirty chunks upload, the flush loop reads them from the same NVMe, and a fraction of requests wait behind that traffic.

Commits stay cheap. This cache drive reports a write-through cache, so flexBlock completes writes durably and the kernel sends no FLUSH or FUA commands: a commit costs its writes and no cache flush.

Validation behind the benchmark

The same build ran a checksum-verified fuzz suite on the S3-backed volume before the measurements, with fio verifying a CRC32C header on every block and zero error counters throughout.

Random sizes and offsets. 4 KiB to 1 MiB writes over 6 GiB, two jobs, read back and verified.
Mixed read/write at 4 KiB to 256 KiB with a verification backlog, four jobs, 60 s.
Trim, write-zeroes, O_SYNC. Trims mixed with writes and verified to read as zeros; native WRITE_ZEROES including an unaligned range across chunk boundaries; 4 KiB O_SYNC writes.
Crash mid-write. The daemon was killed with SIGKILL 12 s into a random write run. The kernel held the queued I/O, the restarted daemon recovered the device, and fio completed and verified every block.
Cold verification. After detach and attach, every region was read back from the bucket and verified.
Database integrity. Both databases were reopened cold from the bucket. PostgreSQL passed pg_amcheck over every table and index with heap-to-index cross-checks, its account, teller and branch balances agreed to the unit, and a further run of 1,014,125 transactions matched its recorded history exactly. MariaDB passed CHECK TABLE EXTENDED on all 16 tables and their checksums matched the ones taken before the restart.

Method

Item	Detail
Host	AWS i7i.4xlarge, 16 vCPUs, 128 GiB, Linux 7.0. One local NVMe drive holds the raw test partition and the flexBlock cache. Object storage is Amazon S3 in the instance’s region.
fio	fio 3.41, io_uring, O_DIRECT, 20 GiB region, 5 s ramp then 30 s per row; both devices written once sequentially before the read tests. Warm: region written through flexBlock and fully flushed, flush loop active. Cold: daemon stopped, cache detached and re-attached empty, daemon restarted; the sequential reads cover the whole region once.
pgbench	PostgreSQL 18 on XFS, data and WAL on the device under test, page checksums on, scale 500, 32 clients on 16 threads, `shared_buffers` 2 GB, `wal_level` minimal, TPC-B 120 s, select-only 60 s.
sysbench	sysbench 1.0.20 OLTP against MariaDB 11.8 on the same XFS, 16 tables of 1 M rows, 32 threads, 2 GB buffer pool, redo flushed on every commit, O_DIRECT data files; read/write 120 s, read-only and write-only 60 s.
flexBlock	flexBlock 1.0.0 defaults (2 MiB chunks, zstd, zero copy, one queue per physical core), a 200 GiB volume on a 250 GiB cache partition.