Postmortem — Truncated Block Vote and Missing Self-Recovery¶

Docs-site note (2026-07-15): Published per the discipline in Reliability & Conformance. This one is deliberately published with an honest partial status — the triggering defect is fixed and gated, but a broader capability gap it exposed is not yet closed. Stating that plainly is the point of publishing postmortems at all.

Status: Partially resolved. The specific defect that caused the node to accept and vote an incomplete block is fixed, proven, and gated. A broader capability — automatically detecting and repairing onto the correct version of a block after a divergence like this one, instead of requiring a manual recovery — is a named, queued follow-on project and is not yet shipped. This page states that gap directly rather than implying it's closed.

One-line summary¶

Under heavy repair/catch-up load, the node's shred-assembly logic declared a block "complete" after receiving only about half of it (30 of 64 expected ticks, roughly two-thirds of the transactions), because incoming data was being dropped under memory pressure and a partial data set was mis-detected as complete. The node froze and voted on this truncated block. A completeness check existed but ran too late — after the vote had already been sent — and once the divergence occurred, the node had no mechanism to detect that a different, correct version of the block existed and switch onto it automatically. Recovery required a manual restart from a known-good snapshot.

Symptom¶

Same on-chain signature as the header-only realloc incident: votes landing but failing with the vote program's slot-hash-mismatch error, starting from a specific slot and continuing every slot after. This time, a live vote-rejection watcher (see Reliability & Conformance) caught it — it identified the first wrong slot within roughly 30 seconds of the divergence occurring, a substantial improvement over relying on delinquency showing up in an epoch-level credit tally.

What it was not¶

Not equivocation by the block's leader. The leader for the slots around this incident produced exactly one version of the block; there is no evidence anywhere in the node's logs of the leader broadcasting two conflicting versions (zero duplicate/equivocation markers across the entire incident window). The cluster had one canonical block. The node's own copy of it was incomplete.
Not an execution or sysvar bug. The block in question contained only vote transactions (no other program activity), which narrowed early hypotheses toward the vote-processing and clock-sysvar code paths that had been implicated in earlier, unrelated incidents. Those paths turned out to be irrelevant here — the node executed the bytes it had correctly; the bytes it had were simply the wrong (incomplete) set.

Root cause¶

Three separate gaps combined to produce this incident:

False "block complete" signal under load. During a period of heavy repair/catch-up activity, the node was also shedding incoming network frames before they could be parsed, due to a resource-reservoir constraint under memory pressure. The shred-assembly logic detected what it interpreted as a final/completing shred at a data-shred index partway through the block and declared the slot complete — with only 30 of the block's 64 expected ticks and roughly two-thirds of its transactions actually assembled. The chain-linkage check that ran at this point verified that the block chained correctly onto its parent, but did not verify tick count or completeness — so a truncated-but-correctly-chained block passed it.
The completeness check that did exist ran too late. A tick/completeness check did observe the deficit (30 ticks seen vs. 64 expected) — but it was wired as a "deferred" check that logged the deficit and queued it for later enforcement, rather than blocking the freeze. The bank froze and the vote was sent; the deficit was only formally enforced (marking the slot dead) about 727 milliseconds after the vote had already shipped — too late to prevent the divergent vote. The canonical client's equivalent check runs before a block is allowed to freeze at all: an incomplete block is marked dead at replay time and is never frozen or voted on.
No automatic recovery onto the correct version. After the slot was retroactively marked dead, the node had no path to fetch and replay the actual canonical version of that slot and reset itself onto it. Its fork-choice logic was also observing gossip votes from other validators throughout this window — it could see that stake was voting for a different fork — but never counted that observation toward arming a switch away from its own (wrong) version, because its fork-choice view didn't register a competing fork as existing in the first place. With no repair-and-switch path and no armed fork-choice signal, the node's local state kept advancing on its own divergent version of the chain while its on-chain votes kept failing. Only a manual restart from a known-good snapshot recovered it.

The fix (deployed)¶

The first gap — the false completion signal and its late enforcement — is fixed:

The completeness check is now a blocking, pre-freeze gate: a slot that has not received its full expected tick count is marked dead at replay time and is refused before it can freeze or be voted on, matching the canonical client's behavior. This closes the mechanism at its source rather than only tightening the enforcement timing.
Block completion now additionally requires a contiguous shred set (no gaps from the first shred index through the last), removing the specific hole-masking pattern that produced the false "complete" signal under packet loss.

Together these mean the exact trigger observed in this incident — a partial block silently accepted as complete under load — is closed.

Proof¶

Deterministic offline reproduction. Replaying the captured incident window with the fix in place, the node now blocks and marks the slot dead pre-freeze instead of freezing and voting on the truncated block — zero freezes of the incorrect bank hash across the reproduction.
No regression. Roughly 11,000 surrounding slots outside the incident window replay byte-identical to before the fix, and the general offline golden-replay range is unaffected.
Detection performance confirmed. The live vote-rejection watch correctly identified the first divergent slot within about 30 seconds during the real incident — validating that layer of Reliability & Conformance under a genuine failure, not just in testing.
Confirmed a second time, in production, on a structurally similar block. A later block hit the same truncation class under similar load conditions. This time the fix was in place: the pre-freeze gate correctly refused the incomplete block before it could be voted on, and the node stalled safely on that slot instead of freezing and voting a wrong hash — the difference between "safe stall, no bad vote" and this incident's "false vote, delinquent" is exactly the fix described above.

Status and residual risk — stated plainly¶

Closed: the specific mechanism that produced this incident (a truncated block falsely marked complete under repair/packet-loss pressure, voted before the completeness check could block it) cannot recur in its observed form — the gate is now pre-freeze and blocking, and shred contiguity is enforced.

Still open: the third gap — the node's inability to detect that a different, correct version of a block exists after a divergence and automatically repair-and-switch onto it — is not yet fixed. It is scoped as a queued, multi-stage project (a near-term stage that converts a worst-case "node roots a divergent fork past consensus" scenario into a fail-safe stall that gets detected in seconds rather than requiring a full self-heal, followed by the actual repair-and-revive and fork-choice-arming work). Until that project ships, a divergence from a different underlying cause that reaches the same "wrong frozen bank, no automatic path back to canonical" state would still require a manual recovery, the same as this one did. This is exactly the kind of open item the Roadmap and this postmortem discipline are meant to surface rather than paper over.

Lessons¶

A check that observes a problem but doesn't block on it isn't a safety gate — it's a log line. The deferred/shadow completeness check "worked" in the sense that it correctly identified the deficit; it did not work in the sense that mattered, because it ran after the unsafe action (freezing and voting) had already happened. The fix wasn't adding new detection logic — the detection already existed — it was moving the existing check to run before the point of no return.
Detection and recovery are separate capabilities, and both have to exist. Fast detection (the vote-rejection watch firing in ~30 seconds) is a real, demonstrated strength; it does not by itself close the incident if there's no automated path from "detected" to "recovered." Publishing this gap explicitly, rather than only publishing the parts that are fully fixed, is intentional.
Resource pressure (memory/packet-shedding) can turn a normally-benign repair/catch-up scenario into a correctness bug, not just a performance one. Any code path that behaves differently under load deserves the same scrutiny as a pure logic bug.