VexLedger — Vexor's Zig-Native Blockstore¶

Docs-site note: VexLedger is Vexor's blockstore, covered here end to end — format, column-family fidelity, and configuration. For the why behind choosing an append-segment log over RocksDB, see Why an Append-Segment Ledger.

Overview¶

VexLedger is Vexor's blockstore — the on-disk database that stores every shred (the ~1228-byte packets a Solana block is split into), the per-slot metadata, and the transaction/RPC indexes a validator needs. It is the component that Agave implements on top of RocksDB and that Firedancer implements as its own custom fd_blockstore.

VexLedger is a purpose-built, 100% Zig, append-segment store. It is a behavioral drop-in for Agave's RocksDB blockstore — the rest of the validator (replay, repair, RPC) calls the same operations and gets the same answers — but underneath it is a fundamentally different, simpler, and more space-efficient design.

It is live on testnet today: the voting node runs with VexLedger as its blockstore, stays bank-exact under load, and adds negligible overhead. Every interop-facing record is validated byte-for-byte against the Agave reference implementation, currently anchored to the newest testnet release (v4.2.0-beta.0) — zero known drift (see Cross-client fidelity).

Why we built it — the RocksDB problem¶

Agave stores its blockstore in RocksDB, a C++ log-structured-merge-tree (LSM) database. RocksDB is powerful and battle-tested, but for a validator's specific workload — a firehose of small, write-once, slot-ordered shreds with a rolling retention window — it carries well-documented baggage that the Solana validator community has complained about for years:

Write amplification. An LSM tree writes data, then rewrites it again and again as background compaction merges and re-sorts SST files down the level hierarchy. A shred that is written once can be physically copied many times before it is finally pruned. For a write-once workload this is pure waste — disk bandwidth and SSD endurance spent re-sorting data that never changes.
Compaction stalls. Compaction runs on background threads and competes with the validator for disk I/O and CPU. Operators have repeatedly seen compaction bursts cause latency spikes and, in bad cases, replay/vote hiccups — exactly when the node can least afford it.
Disk-usage blowup. Between the level overlap, tombstones, and deferred compaction, on-disk size can balloon well past the logical data size, and reclaiming space after --limit-ledger-size pruning is not instant — deletes become tombstones that themselves must be compacted away.
Tuning pain. Getting RocksDB to behave (write buffers, level sizing, compaction style, rate limits) is a deep, fragile art, and the knobs interact. It is a recurring source of operator toil.
Not Zig, and not ours. RocksDB is a large C++ dependency. Vexor's north star is a fully Zig-native, auditable, in-house validator; a giant C++ LSM at the storage core is the opposite of that.

Firedancer reached the same conclusion and abandoned RocksDB for its own custom storage (fd_shredb / fd_store), explicitly to escape compaction overhead and background-thread/sandbox friction. We took the same lesson — and went one step further by making ours persistent and crash-recoverable (Firedancer rebuilds shreds from the network on restart; we keep them).

Why not just use an off-the-shelf embedded DB?¶

We evaluated the plug-and-play options and rejected each for this workload:

Option	Why not
RocksDB / Speedb / TerarkDB / LevelDB	C++ LSM — the write-amp/compaction baggage above, plus a non-Zig dependency.
`rocksdb-zig` (what the Sig validator uses)	A Zig binding — still RocksDB underneath, inherits all of the above.
TigerBeetle's storage engine	Fixed double-entry-accounting schema, no arbitrary delete — wrong shape.
redb / sled / fjall / ParityDB	Rust crates — a Rust FFI dependency, against the Zig-native goal.

For a workload that is write-once, slot-ordered, and pruned as a rolling window, the right structure is not a general-purpose LSM at all. It's an append-only segment log — which is what we built.

What's in it — architecture¶

VexLedger combines Firedancer's storage discipline (fixed-size, append-only, no LSM, O(1) hot operations) with Agave's behavioral contract (the same column families and API the validator expects) and adds the one thing Firedancer deliberately skips: persistence and crash recovery.

Storage layer (on disk)¶

segment files:  vexledger-<seq>.seg   — append-only. Shreds/meta/roots are appended
                sequentially as they arrive. Each record = [17-byte header | payload].
                A segment is sealed (made read-only) once it reaches ~256 MiB, and the
                NEXT slot starts a fresh segment (so slots stay whole-in-one-segment).
                → Data is written ONCE and never rewritten. There is NO compaction.

Because the file is append-only, a shred is written exactly once. There is no background process re-sorting it. This is the structural reason VexLedger has ~1× write amplification versus RocksDB's multi-× LSM overhead.

Index layer (in memory, rebuildable)¶

A set of in-memory maps point into the segment files, so every lookup is O(1) and never blocks on disk geometry:

shred_index / code_index: (slot, index) → {segment, offset, len}
slot_meta: slot → SlotMeta (mirrors Agave's fields)
erasure_meta, merkle_root_meta: FEC-set bookkeeping + duplicate detection
roots: the set of rooted slots
sig_index: signature → slot (RPC point lookups)
lowest_cleanup_slot: the pruning watermark

The index is rebuilt from the segment files on boot, so it is never the source of truth — the segments are.

The column families¶

VexLedger implements the full set of interop-load-bearing Agave column families, with the exact on-the-wire encoding of the anchored Agave release (v4.2.0-beta.0) for each:

Family	Key	Value encoding
`data_shred`, `code_shred`	`(slot u64 BE, index u64 BE)`	raw shred bytes
`meta`	`slot u64 BE`	wincode `SlotMetaV3`
`erasure_meta`	`(slot, fec_set_index u64 BE)`	wincode `ErasureMeta`
`merkle_root_meta`	`(slot, fec_set_index u32 BE)`	wincode `MerkleRootMeta`
`bank_hashes`	`slot u64 BE`	wincode `FrozenHashVersioned` (37 B)
`transaction_status`	`(signature, slot u64 BE)`	protobuf `TransactionStatusMeta`
`rewards`	`slot u64 BE`	protobuf `Rewards`
`address_signatures`	`(pubkey, slot, tx_index u32 BE, sig)`	`bool`
`root`, `transaction_memos`, `blocktime`, `block_height`	slot / sig	bincode scalars

Plus a set of Vexor-native record kinds that have no Agave equivalent and power capabilities other clients don't have: a transaction-wire store, a slot→signature index, a per-slot blockhash record, and the flight recorder (see moats).

Reads, writes, pruning, recovery¶

Writes append to the active segment and update the in-memory index. fsync cadence is one per completed slot by default (tunable — see Configuration).
Reads return a slice straight out of the segment file (zero-copy shred-serve to repair and replay).
Pruning (--limit-ledger-size semantics): keep a bounded recent window; evict the oldest whole sealed segment with a single unlink(). This is O(1) and reclaims space instantly — no compaction copying live data, the opposite of RocksDB's tombstone-then-compact dance.
Crash recovery: on boot, read the manifest, rebuild the index from the surviving segments, and replay the active segment's tail (truncating any torn final record). Deterministic and bounded.

How it's different¶

	Agave (RocksDB)	VexLedger
Structure	LSM tree	Append-only segment log
Write amplification	Multi-× (compaction rewrites)	~1× (write-once)
Background work	Compaction threads compete for I/O	None — no compaction at all
Space reclaim on prune	Tombstone → later compaction	O(1) `unlink()` of a whole segment
Language / deps	C++ (large external dep)	100% Zig, std-only
Tuning surface	Deep, fragile (many interacting knobs)	A handful of clear env knobs
Crash recovery	RocksDB WAL	Rebuild index from segments + tail replay

The design borrows the best of both existing clients and adds one thing neither blockstore has: Firedancer's no-LSM, append, O(1) discipline, Agave's behavioral + byte-level contract, and persistence with offline replay on top.

Why it's better¶

Space-efficient by construction. Write-once + whole-segment eviction means ~1× write amplification and instant space reclaim, with no compaction overhead and no disk blowup. This was a stated priority, and it falls directly out of the architecture rather than out of tuning. (Steady-state write-rate figure TBD — flagging rather than citing an unverified number.)
No compaction means no compaction stalls. There is no background thread contending with replay/vote for disk I/O. The persist path runs on a dedicated cold core, off the consensus path entirely.
Crash-safe and offline-replayable. Because shreds are persisted and recoverable, any rooted slot can be re-replayed offline, with no network — and reproduces the cluster's bank hash exactly. (VexLedger-specific replay sample size TBD — flagging rather than citing an unverified number; see the byte-fidelity methodology for the cited 1992-canonical-slot offline golden replay that covers the consensus path.) This re-replay capability is the foundation of the moats below.
Cross-client byte-fidelity. Every interop value is validated byte-for-byte against the Agave reference implementation, currently anchored to v4.2.0-beta.0 (see next section), so the validator behaves identically and RPC consumers get byte-identical responses.
Auditable and ours. It is a few thousand lines of straightforward Zig with golden-vector tests, not a million-line C++ dependency — fully in scope for a clean-room, independently-auditable client.

Beyond parity — the moats¶

Because VexLedger persists shreds and can re-replay them, it unlocks two capabilities no other validator client has:

Flight recorder. At every block freeze, VexLedger captures the consensus-critical fingerprint of the slot (bank hash, parent hash, signature count, PoH hash, accounts lt-hash) into a compact per-slot record. This is a black box for the validator's consensus state.
Real-time divergence alarm — armed live. Combining the flight recorder with the offline-replay engine, a background alarm thread detects the first slot where the node's bank hash diverges from the cluster and automatically re-replays it offline, narrowing the cause to one of the bank hash's four inputs (see Consensus). It is wired live and running in production, non-blocking (a single enqueue on the replay thread; all RPC/replay/comparison work happens on a separate thread), currently in its soak period — see Reliability & Conformance for the honest status of what it has and hasn't caught yet.

The flight recorder and the divergence alarm are both opt-in via environment flag (see Configuration below); in the current live deploy both are enabled. Off, they have zero effect on the consensus path.

Extraction as a public good¶

The generic storage core (an append-segment, bounded, crash-safe, std-only Zig store) is specified for extraction as a standalone open-source Zig package — zseglog — with the Solana-specific column families layered on top. It fills a real gap in the Zig ecosystem (a maintained embeddable KV/segment store) and is a planned shipped public good.

Cross-client fidelity¶

A validator's blockstore is only "correct" if it stays in line with the other validated clients — the column family names, key byte-layouts, and value encodings have to match the canonical implementation exactly, or cross-client tooling breaks.

VexLedger's interop-facing records are validated byte-for-byte against the Agave reference implementation, currently anchored to the newest testnet release (v4.2.0-beta.0 — Vexor re-anchors this audit to each new release as it ships), with a verdict of zero known drift: every column-family name (exact snake_case), every key (big-endian, correct field widths and order), and every value codec (wincode for the meta families, protobuf for transaction_status and rewards) is byte-for-byte faithful, and the RPC JSON output byte-matches the reference. The known encoding traps are all handled correctly (the ErasureMeta.fec_set_index u64-on-disk vs MerkleRootMeta u32 split; ShredIndex as a bit-vector not a set; the 37-byte FrozenHashVersioned; the 17-tag TransactionStatusMeta including cost_units; protobuf commission rendered as a JSON number, not a string).

One important clarification on what "compatible" means. VexLedger's on-disk format is its own append-segment layout — it is not a RocksDB database, so agave-ledger-tool cannot open the .seg files directly (exactly as it cannot open Firedancer's fd_blockstore). The compatibility guarantee is behavioral and at the wire boundary: the validator behaves identically, the RPC responses are byte-identical to Agave, and the export-form keys and values are RocksDB-/ledger-tool-compatible. That is the right and expected model for a purpose-built blockstore.

Configuration¶

VexLedger is controlled by environment flags. The storage path is always active when the ledger is enabled; the richer capabilities are opt-in and dormant by default (zero overhead, bank-hash-neutral when off).

Flag	Effect	Default
`VEX_LEDGER=1`	Enable the VexLedger write path (shreds + metadata).	on (live deploy)
`VEX_LEDGER_MAX_BYTES=<n>`	Bound the on-disk ledger to `<n>` bytes (`--limit-ledger-size`); evict oldest whole segments.	baked to `107374182400` (~100 GiB) if unset
`VEX_LEDGER_KEEP_SLOTS=<n>`	Alternative: keep the last `<n>` slots. (Bytes takes precedence if both set.)	baked to `216000` if unset
`VEX_LEDGER_FSYNC_EVERY=<n>`	fsync once per `<n>` completed slots (1 = every slot, the durable default; higher = more throughput).	1
`VEX_LEDGER_CONTENT=1`	Capture full transaction content for archival RPC (getBlock/getTransaction).	off
`VEX_LEDGER_FLIGHT=1`	Enable the flight recorder (prerequisite for the divergence alarm).	off (enabled in the current live deploy)
`VEX_DIVERGE_ALARM=1`	Enable the real-time divergence alarm's freeze-tap and background comparison thread.	off (enabled in the current live deploy)

The build flag -Dvex_ledger=true compiles the subsystem in; with it absent, VexLedger is entirely absent from the binary (zero-cost).

Status (2026-07-11)¶

Live on testnet, voting bank-exact, as the node's blockstore. Negligible overhead; ~1× write amplification.
Agave byte-fidelity (anchored to v4.2.0-beta.0): validated, zero known drift, both storage and RPC-read halves.
Offline replay: proven bank-exact on rooted slots (VexLedger-specific sample size TBD — flagging rather than citing an unverified number; the cited byte-fidelity methodology figure is 1992 canonical slots across an epoch boundary for consensus changes).
Pruning, telemetry, and tunable fsync: implemented and gated; deployed on the next maintenance build.
Real-time divergence alarm: armed live, non-blocking, soaking in production (see Reliability & Conformance).
Roadmap: the standalone zseglog extraction (specified), and archival-mode read optimizations (mmap zero-copy reads).