Reliability & Conformance¶

Docs-site note (2026-07-16): This page is the operational companion to Why Byte-Faithful Parity and Parity & Fidelity Audit. Those pages argue why strict byte-for-byte parity is the only coherent target for a consensus client and lay out the proven results. This page is about the machinery that catches it when parity slips anyway — the gate battery that has to pass before anything reaches the live node, the detection layer that watches the live node itself, and the discipline applied when detection fires. See Incident Postmortems for the write-ups this machinery has actually produced, and How We Track Upstream for how the target itself is kept current.

Why this page exists¶

A hash-based consensus client either matches the cluster's bank hash or it doesn't — there is no "mostly correct" state (see Why Byte-Faithful Parity for why). Given that, reliability isn't just "does it stay up" — it's a chain of guarantees:

Most divergences never reach the live node, because they're caught by a gate before deploy.
The ones that aren't caught are detected fast, on the live node, rather than discovered hours later as unexplained delinquency.
When one is found, the response preserves the evidence needed to find the real root cause, fixes it with a named mechanism (not a guess), and turns the exact failure into a permanent regression test.

This page documents that chain honestly, including where it has actual gaps today.

The gate battery — what has to pass before anything ships¶

Every change that can affect consensus or on-disk/wire records has to clear a stack of gates, cheapest first:

Gate	What it checks	Scope
Known-answer tests (KATs)	Function-level output vs. byte-exact vectors extracted directly from Agave's own crate internals (not hand-written expectations)	Codecs, syscalls, proof programs, instruction handlers — hundreds of vectors, including dormant/feature-gated code that hasn't activated on-cluster yet
Offline golden replay	Full-node offline re-replay of a pinned, cluster-attested range of canonical slots (1,992 slots spanning a full epoch boundary), asserting the resulting bank hashes are byte-identical to the live cluster's	Every commit that touches the consensus or runtime path
Vote-instruction differential comparison	Instruction-for-instruction comparison against a large capture of real, live vote traffic	Vote-program changes specifically (see The Vote Program)
Incident-specific regression gates	The exact slot range and conditions of a past divergence, re-run and asserted against the canonical hash, not just "doesn't crash"	Added permanently after every confirmed incident — see Incident Postmortems

None of these are optional or skippable for a consensus-path change. The offline golden replay in particular is the same instrument used to find several of the incidents documented below — it is both the pre-deploy gate and the reproduction tool.

The golden-vector approach also means correctness for a feature can be proven before the network ever activates it, rather than discovered the day it flips on — see Parity & Fidelity Audit → SVM crypto syscalls for a concrete example of a dormant code path with full KAT coverage.

A fifth gate in progress: fixture-based conformance testing¶

A separate effort, still in progress and not yet wired into the gate battery above, runs Vexor's sBPF VM and native program dispatch against a large third-party fixture corpus (47,240 instruction-level test vectors covering native programs, the BPF loader family, precompiles, and ZK proof verification) to find behavioral bugs mechanically, one fixture at a time, instead of one production incident at a time. The runner scores itself honestly: 93.64% of the 16,897 fixtures currently in scope pass (gate threshold ≥80%, cleared by 13.64 points, 2026-07-16). The remaining 30,343 fixtures are excluded as known, named gaps — precompile verification and BPF-loader management instructions the harness can't reach yet, and roughly twenty vote-program instruction variants not yet implemented — reported separately, not folded into the percentage.

An earlier pass on this same corpus scored 79.05% (below the 80% gate) and surfaced three concrete, named findings: a missing account-owner precondition check in the vote program's instruction dispatch, a spurious PDA-marker rejection in the system program's TransferWithSeed handling, and a compute-unit-accounting gap on BPF-path failures. All three are now fixed, verified against the same fixture corpus with byte-identical repeat runs, and the fixes dropped the non-known-gap bug count from 2,591 to 436 fixtures — which is what moved the score from 79.05% to 93.64% and cleared the gate. The 436 remaining bugs and 639 harness-mismatch fixtures are disclosed, not hidden: mostly a residual vote-program result-shape gap on correctly-owned accounts, a small UTF-8-validation edge case the marker fix exposed, and two narrow compute-unit-model precision gaps. This tool is real, finding real bugs, and now clears its own gate — but it is still not wired into the mandatory gate battery above, so it is not yet a claim that Vexor's consensus path is validated by it.

Live detection — closing the gap between "wrong" and "known wrong"¶

Gates catch what they're written to check. A gate battery can only be as good as the incidents and edge cases it already knows about — which is exactly why detection on the live node matters: it's the backstop for the case the gates didn't anticipate.

Vote-rejection watch¶

The fastest and simplest signal is on-chain: a vote transaction can land but still be rejected by the vote program with a specific error that means "the block hash you're voting on disagrees with the network's SlotHashes sysvar" — i.e., the node computed a bank hash for a canonical block that doesn't match what the rest of the cluster computed for the same block. That is a direct, unambiguous signal of a bank-hash divergence, and it doesn't require waiting for the node to fall behind and show up as delinquent in an epoch-level credit tally.

A watcher process polls the node's own vote outcomes for this exact failure signature. In its first real-world firing (documented in Incident Postmortems), it identified the first wrong slot within about 30 seconds of the divergence occurring — versus the alternative of a human noticing delinquency, which in an earlier, less-instrumented incident took on the order of tens of minutes.

Divergence alarm — armed live¶

A more ambitious detector is built on top of the blockstore's flight recorder — VexLedger already persists a consensus fingerprint for every slot (bank hash, parent hash, signature count, PoH hash, accounts lattice hash; see VexLedger). At every bank freeze, a single non-blocking enqueue hands the slot's {slot, bank_hash} pair to a background alarm thread — the replay thread itself never performs RPC, re-replay, or any blocking work, so the alarm cannot perturb bank hash or consensus timing. The alarm thread compares each entry against a cluster RPC oracle and, on a mismatch, automatically triggers an offline re-replay of the suspect slot and narrows the divergence to one of the bank hash's four inputs (parent hash, signature count, PoH hash, or the accounts lt-hash) using the same reusable classification engine that has driven past manual incident investigations (see Incident Postmortems).

Status, honestly: the alarm is wired live and running in production, non-blocking and soaking. It has known-answer test coverage for its classification logic and a proven offline-replay engine underneath it, but as of this writing it has not yet fired against a real production divergence, so its live auto-replay path is proven by design and by test, not yet by an actual catch. Until it has a production track record of its own, the vote-rejection watch above — which has a real-world track record (see the postmortems) — remains the detection layer this page leans on for "actually observed working."

Why divergence recovery is a deliberate action, not an automatic one¶

Vexor's in-process liveness watchdog (see Deploy → supervision and restart) restarts the process on a genuinely hung node — a real liveness failure. A bank-hash divergence is a different animal: the process is not hung, it is running fine and computing a wrong answer with total confidence. Auto-restarting into a fresh snapshot the moment that's suspected would "fix" the symptom (the node comes back current) while destroying the one thing that lets you find and fix the actual bug — the divergent in-memory and on-disk state at the moment it happened.

So the operational rule for a suspected divergence is: capture the evidence first, then recover. In practice that means preserving the frozen bank state and the ledger segments around the divergent slot before restarting the node from a known-good snapshot. This is slower than an automatic respin, and it is intentional — every postmortem below exists because this discipline was followed instead of silently restarting past the problem.

Incident postmortem discipline¶

When the gates and live detection above do catch something, the investigation and fix follow a fixed process:

Capture first. Freeze the evidence (bank state, ledger segments, logs) before anything else happens to the node.
Reproduce deterministically, offline. A hypothesis about the root cause doesn't count until it reproduces the exact incorrect hash offline, with no network involved, from the captured state. "Plausible" is not the bar; bit-identical reproduction of the wrong answer is.
Name the mechanism. The fix ships against a specific, named code path and mechanism — not a pattern match against a previous incident that merely looks similar. Superficially similar symptoms have turned out to have unrelated causes more than once; each hypothesis is individually falsified or confirmed against the evidence rather than assumed by resemblance.
Re-verify, don't assume. Prior fixes that are structurally related to the new incident are re-checked against the current code, not presumed to still hold — upstream and local code both move, and a fix that was correct once can be silently undone by an unrelated later change.
Fix forward, gate permanently. The fix is applied as a forward change (never a revert to a prior state), proven against the general golden-replay range so it introduces no regression, and the exact incident conditions become a permanent, named regression gate — see the gate battery above.
Publish the postmortem. See Incident Postmortems for the write-ups this discipline has produced so far, including the ones where the fix is complete and the ones where it is honestly partial.

Vote-credit reliability — what the numbers actually mean¶

Vexor's testnet node earns on the order of ~98.7% of the maximum available vote credits, measured against the theoretical 16-votes-per-slot ceiling (see Parity & Fidelity Audit for the full context). It's worth being precise about what that number is and isn't:

When the node is current, it lands votes at the full per-slot rate — the shortfall is not a background rate of partial or missed votes during normal operation.
The gap is downtime, concentrated in the windows between a divergence being detected and the node being recovered onto a known-good state. Every incident in Incident Postmortems is a concrete example of exactly that kind of window, along with what caused it and what changed afterward to prevent that specific cause from recurring.
That framing is why this page and the incident write-ups exist as public documentation rather than being kept as internal notes: the reliability story for a hash-exact client isn't "we never had a problem," it's "every problem we had is named, reproduced, fixed with a permanent gate, and written down."