Why an Append-Segment Ledger¶

Docs-site note (2026-06-25): A "why we built it this way" page — the decision and tradeoffs only; the full design lives in VexLedger (Blockstore).

The decision¶

Every validator needs a blockstore — the on-disk database that holds every shred, the per-slot metadata, and the transaction indexes. Agave builds this on RocksDB. Firedancer wrote its own custom store. Vexor wrote its own too: a 100% Zig, append-only segment log.

This page explains why we did not reach for an existing database. The short version: a validator's blockstore workload is write-once, slot-ordered, and pruned as a rolling window — and for that shape, a general-purpose log-structured-merge (LSM) database is the wrong structure. The right structure is an append-only log, so that is what we built.

For how it works — the segment format, the column families, the byte-level Agave fidelity — see the architecture page.

The RocksDB problem¶

Agave's blockstore is RocksDB, a C++ LSM database. RocksDB is powerful and battle-tested, but for a firehose of small, write-once, slot-ordered shreds it carries baggage the validator community has complained about for years:

Write amplification. An LSM tree writes data, then rewrites it again and again as background compaction merges and re-sorts files down the level hierarchy. A shred written once can be physically copied many times before it is pruned. For a write-once workload that is pure waste — disk bandwidth and SSD endurance spent re-sorting data that never changes.
Compaction stalls. Compaction runs on background threads and competes with the validator for disk I/O and CPU. Operators have repeatedly seen compaction bursts cause latency spikes — exactly when the node can least afford it.
Disk-usage blowup. Between level overlap, tombstones, and deferred compaction, on-disk size can balloon well past the logical data size, and reclaiming space after pruning is not instant: deletes become tombstones that themselves must be compacted away.
Tuning pain. Getting RocksDB to behave (write buffers, level sizing, compaction style, rate limits) is a deep, fragile art, and the knobs interact. It is a recurring source of operator toil.
Not Zig, and not ours. RocksDB is a large C++ dependency. Vexor's goal is a fully Zig-native, auditable, in-house validator; a million-line C++ LSM at the storage core is the opposite of that.

Firedancer reached the same conclusion and abandoned RocksDB for its own custom storage, explicitly to escape compaction overhead and background-thread friction. We took the same lesson — and went one step further by making ours persistent and crash-recoverable (Firedancer rebuilds shreds from the network on restart; Vexor keeps them).

Why not an off-the-shelf embedded DB?¶

The obvious next question is: if not RocksDB, why not some other plug-and-play store? We evaluated the options and rejected each for this specific workload.

Option	Why not
RocksDB / Speedb / TerarkDB / LevelDB	C++ LSM — the write-amp/compaction baggage above, plus a non-Zig dependency.
`rocksdb-zig` (what the Sig validator uses)	A Zig binding — still RocksDB underneath, inherits all of the above.
TigerBeetle's storage engine	Fixed double-entry-accounting schema, no arbitrary delete — wrong shape.
redb / sled / fjall / ParityDB	Rust crates — a Rust FFI dependency, against the Zig-native goal.

The common thread: a general-purpose database optimizes for random reads, updates, and deletes over data that changes. A validator's shreds never change after they are written and are dropped in whole slot-ordered batches. Paying LSM overhead to support mutation the workload never uses is the wrong trade.

What we chose instead¶

An append-only segment log. Shreds and metadata are appended sequentially to fixed-size segment files; a segment is sealed once full and never rewritten. The benefits fall directly out of the structure, not out of tuning:

	Agave (RocksDB)	VexLedger
Structure	LSM tree	Append-only segment log
Write amplification	Multi-× (compaction rewrites)	~1× (write-once)
Background work	Compaction threads compete for I/O	None — no compaction at all
Space reclaim on prune	Tombstone → later compaction	O(1) `unlink()` of a whole segment
Language / deps	C++ (large external dep)	100% Zig, std-only
Tuning surface	Deep, fragile (many interacting knobs)	A handful of clear env knobs

Because data is written once and pruning evicts a whole sealed segment with a single unlink(), there is no compaction thread to contend with replay or voting, and space is reclaimed instantly. Persisting the shreds also makes any rooted slot re-replayable offline — a capability neither other client's blockstore offers, and the foundation for Vexor's flight-recorder and divergence-alarm features.

The full architecture, the byte-level Agave-fidelity audit, the column-family table, and the configuration flags are all on the VexLedger architecture page.

Honest caveats¶

Not a RocksDB on disk. VexLedger's on-disk format is its own append-segment layout, so agave-ledger-tool cannot open the .seg files directly — exactly as it cannot open Firedancer's custom store. The compatibility guarantee is behavioral and at the wire boundary: the validator behaves identically and the RPC responses are byte-identical to Agave.
Turnkey ledger-tool export is not yet built (PLANNED). A one-command export that re-materializes the segments into a RocksDB column-family layout an unmodified agave-ledger-tool could open is latent — the keys and values are already export-form-compatible, but the convenience tooling around it is not yet shipped.
The richer capabilities are opt-in. Persistence and the core write path are live; the flight recorder and the real-time divergence alarm are behind environment flags, off by default at the code level, but both are enabled in the current live deploy — the alarm is armed live and soaking, not merely designed. See the architecture page status section for the current state.