Skip to content

Incident Postmortems

Docs-site note (2026-07-15): This is an index of Vexor's public bank-hash divergence postmortems. It exists because of the discipline described in Reliability & Conformance: every confirmed divergence gets a deterministic root cause, a named fix, and a permanent regression gate, and the write-up gets published rather than kept as an internal note. Internal operational detail (hostnames, file paths, credentials, timestamps down to the second, internal tooling names) is deliberately scrubbed from these pages; the technical mechanism, the affected slots, and the fix are not.

What gets published here

This index covers bank-hash divergences — cases where Vexor's testnet node computed a different bank hash than the rest of the cluster for a canonical block, which is the one class of bug that actually threatens the "independent client that stays in consensus" premise the whole project is built on. It does not cover routine downtime, planned maintenance, or non-consensus bugs; those aren't tracked as postmortems because they don't carry the same risk to network health.

Each entry below states plainly whether the underlying issue is fully resolved, resolved for the specific trigger with a known broader gap still open, or still in progress — see Roadmap for how open items are tracked more generally.

Index

Date Slot Class Summary Status
2026-07-13 421,724,293 Execution — silent truncation on account-data growth A third-party program's account-resize pattern hit two stacked bugs in Vexor's sBPF account-growth handling, silently truncating a write the program believed had succeeded. Resolved
2026-07-14 421,935,259 Ingest — false slot-completion under repair load Under heavy repair/catch-up load, the node accepted and voted a truncated copy of a block instead of the full canonical one, then could not self-recover onto the correct version. Partially resolved — the specific trigger is fixed; general self-recovery from this class of divergence is a tracked follow-on
2026-07-15 422,156,256 Execution — one-ULP floating-point rounding gap at an epoch boundary A general-purpose power function used in the epoch-boundary inflation-reward calculation rounded one ULP differently from the reference math library on that epoch's input, flipping the reward pool by one lamport and diverging the bank hash. Resolved

How to read a postmortem here

Each write-up follows the same shape:

  1. Symptom — what an operator would actually observe (the practical "how do I recognize this").
  2. What it was not — hypotheses that were checked and ruled out, so a future investigation doesn't re-walk the same dead ends.
  3. Root cause — the specific, named mechanism, traced to source.
  4. The fix — what changed, and how it was proven correct without regressing anything else.
  5. Status and residual risk — stated plainly, including any part of the problem class that isn't fully closed yet.

See also