Postmortem — Header-Only Realloc Divergence at an Epoch Boundary¶

Docs-site note (2026-07-15): Published per the discipline in Reliability & Conformance. Internal paths, hostnames, and tooling names are scrubbed; the technical mechanism, slot numbers, and account sizes are not — they're what makes this useful to anyone debugging a similar class of bug.

Status: Resolved. Root cause identified, fixed, proven against a deterministic offline reproduction, and gated permanently against recurrence.

One-line summary¶

A transaction that grew an account's size across several instructions in one transaction — a normal, spec-legal pattern — hit two stacked bugs in Vexor's account-data-growth handling. The second (and more subtle) of the two caused a resize that the executing program believed had succeeded to be silently clamped back down at commit time, producing a different account state — and therefore a different bank hash — than the canonical client computed for the same block. Every subsequent vote from the node was rejected on-chain until it was recovered from a known-good snapshot.

Symptom¶

The node went delinquent while local replay kept advancing — banks kept freezing normally, the process was not hung or stuck. getSignaturesForAddress on the node's vote account showed votes landing but failing, with the on-chain vote program's error indicating that the node's computed hash for a slot disagreed with the network's own record of that slot's hash. That specific signal — votes landing, not missing, but failing with that particular error — is what makes this bug class recognizable: it means the node executed a block and produced a confidently wrong answer, not that it fell behind.

A fresh-snapshot restart "cured" the symptom by wiping the divergent state, but that is a recovery action, not a diagnosis — see why divergence recovery is deliberate, not automatic.

What it was not¶

The investigation initially had several plausible-looking hypotheses, given the divergence landed shortly after an epoch boundary (inside the window where per-epoch staking rewards are distributed across many slots). All of the following were individually checked against byte-level evidence and ruled out:

Not the epoch boundary itself, and not a regression in a previously-fixed epoch-boundary bug — the boundary slot and the slots immediately after it were confirmed clean; the divergent slot landed 37 slots later, and the proximity to the boundary turned out to be coincidental.
Not partitioned epoch-rewards distribution — every stake account written in the relevant reward partition was checked byte-for-byte against the canonical chain, in both directions (nothing missing on either side), and the partition-assignment function itself was independently re-derived and matched exactly.
Not a fork or an orphaned block — the node's proof-of-history hash matched the canonical chain's blockhash for every sampled slot, including the divergent one. Same block, same parent linkage, different computed bank hash — i.e., an execution-level divergence, not a replication/consensus-choice problem.
Not a regression of prior, unrelated fixes — several structurally-adjacent fixes (a clock-sysvar anchoring fix, an epoch-transition vote-accounting fix) were re-verified against the current source and confirmed intact and unrelated.

Ruling these out took the bulk of the investigation time. It's included here because "the divergence landed near an epoch boundary" was a strong but ultimately misleading pattern match — the actual defect had nothing to do with epoch transitions.

Root cause¶

The triggering transaction called a widely-used, third-party validator-metadata program (program ID HistoryJTGbKQD2mRgLZ3XhqHnN811Qpez8X9kCcGHoa) with an instruction sequence that creates an account and then grows it across several follow-up instructions in the same transaction, from 10,240 bytes up to 65,856 bytes. This growth pattern is legal per the runtime's account-resize rules and is executed correctly by the canonical client. Two independent defects in Vexor's implementation combined to break it:

The primary carrier — a header-only resize was silently clamped. The triggering program's resize call writes only the account's serialized length field on that particular call path, without also touching the newly-grown region of the account's byte buffer. Vexor's account-commit logic took the minimum of the previous committed length and the new length when writing back — which is correct for a shrink, but for this specific "grow, but don't touch the new bytes yet" pattern it silently discarded the growth instead of committing it and zero-filling the new region (which is what the canonical client does). The executing program had no way to observe this — from its point of view, the resize succeeded.
A secondary, independently-wrong budget constant. Vexor's cap on how much an account's data can grow within a single transaction was set to the same numeric value as the correct per-instruction cap (10,240 bytes) — a copy-paste-shaped error, off by roughly three orders of magnitude from the canonical per-transaction limit. This alone did not reproduce the divergence (proven empirically — fixing only this constant and re-running the deterministic reproduction still produced the wrong hash), but it was a real defect independent of the primary carrier and was fixed alongside it.

The net effect: the account that should have grown to 65,856 bytes and roughly 459.2M lamports stayed stuck at 10,240 bytes, a second account the canonical execution wrote was never written on Vexor's side at all, and the transaction's fee payer ended up with a different lamport balance than canonical — while the transaction itself still reported success on both clients. "The transaction succeeded" and "the transaction executed identically to canonical" turned out to be different claims.

The fix¶

Corrected the per-transaction data-growth budget constant to match the canonical value (the correct constant already existed and was used correctly elsewhere in the codebase for a related check — this was a matter of routing the transaction-level check to the right constant, not inventing a new one).
Changed the account-commit path so a valid grow (bounded by the per-instruction cap, the per-transaction budget, and an absolute per-call ceiling) commits the new length and zero-fills the newly-grown region, instead of clamping it back down — matching the canonical client's zero-fill-on-resize behavior.
Added targeted known-answer tests covering: the corrected budget constant, the exact incident shape, a pre-fix witness test (so the test suite can prove it would have caught the bug had it existed sooner), that the per-instruction cap still binds independently of the per-transaction budget, and that a genuinely over-budget grow is still correctly refused.

Proof¶

Deterministic offline reproduction. The exact incident window was replayed offline, with no network involved, from evidence captured at the time of the incident. The pre-fix binary reproduces the incorrect bank hash bit-for-bit; the fixed binary reproduces the canonical bank hash for the same slot, with the surrounding slots unchanged, the affected account correctly grown to 65,856 bytes, and the second, previously-missing account correctly written.
No regression. The general offline golden-replay range (1,992 canonical slots spanning a full epoch boundary) was re-run against the fix and remains byte-identical to canonical.
Independent audit. The fix was reviewed against the incident evidence before deployment.

This deterministic reproduction is now kept as a permanent, named regression gate — see the gate battery.

Status and residual risk¶

Resolved and deployed. The fix has been live since shortly after the incident, cleared all gates above, and the specific defect class (silent-clamp-on-grow semantics, and the mismatched budget constant) is closed. Two narrow, documented residuals remain and are tracked as ordinary follow-on work, not open divergence risk: the runtime currently clamps an over-budget grow rather than erroring the instruction the way canonical does (the clamp-vs-error distinction only matters for a transaction that was already going to fail differently), and the shrink-side accounting for this same code path has a known minor gap. Neither has been observed to produce a bank-hash divergence.

Lessons¶

Location is not cause. A divergence landing inside a pattern-matching window (here: near an epoch boundary, inside a rewards-distribution range) strongly suggested a rewards bug. It wasn't one. The byte-level exoneration process — checking claims in both directions, not just confirming the first plausible story — is what actually found the real cause.
"The transaction succeeded" is not "the transaction executed identically to canonical." Any place a runtime silently clamps, truncates, or best-effort's a write instead of committing it exactly or erroring loudly is a latent divergence. The general engineering takeaway carried forward: prefer matching canonical's error behavior over silently downgrading a write.
Ground truth needs full write-set visibility, not just the parts a wallet or block explorer surfaces. Some of the account writes involved here don't show up in a standard "rewards paid" view at all; finding them required an independent, full-account-set comparison against a canonical replay.