Postmortem — Epoch-990 Inflation-Rewards Floating-Point Divergence¶

Docs-site note (2026-07-16): Published per the discipline in Reliability & Conformance. Internal paths, hostnames, and tooling names are scrubbed; the technical mechanism, slot numbers, and the exact numeric values are not — they're what makes this useful to anyone debugging a similar class of bug.

Status: Resolved. Root cause identified down to the exact bit, fixed, proven against a deterministic offline reproduction that reproduces the canonical bank hash exactly, and gated permanently against recurrence.

One-line summary¶

At the first slot of a new epoch, the validator's inflation-reward calculation used a general-purpose floating-point power function whose last-bit rounding differed from the reference C math library by exactly one unit in the last place (ULP) on that epoch's specific input. That one-ULP difference flipped the computed epoch reward pool by exactly one lamport, which changed the bytes of an on-chain sysvar that all subsequent accounts in the epoch derive from — and therefore produced a different bank hash than the rest of the cluster for every slot after the boundary.

Symptom¶

The node went delinquent immediately at the epoch boundary. Unlike a stall, replay kept advancing normally — the node froze banks and voted on every subsequent slot — but every vote failed on-chain with the vote program's slot-hash-mismatch error, the same recognizable signature as prior divergence incidents on this project (see the other postmortems): a confidently wrong answer, computed and voted with no local sign of trouble.

What it was not¶

Not a feature-activation gap. No network feature activated at this epoch boundary or either of the two surrounding it; the divergence was not explained by a newly-enabled code path.
Not a gross inflation-rate bug. An early comparison against a stale, hand-calibrated reference value in the node's own diagnostic logging suggested the reward rate itself might be wrong by a meaningful margin. That reference value turned out to be a stale constant left over from a much earlier epoch (the real reward rate tapers every epoch, and the stale log line hadn't been updated to reflect that) — a logging artifact, not a calculation bug. Recomputing the expected rate from first principles showed the rate was correct to many significant digits; only the very last bit of the final reward-pool value was wrong.
Not a shred, network, or Proof-of-History problem. The node's Proof-of-History hash for the boundary slot matched the canonical chain's blockhash exactly. This was an execution-only divergence — same block, same input, different computed output — not a replication or fork-choice issue.
Not a repeat of the prior epoch-boundary incident (see the header-only realloc postmortem) — that was an account-data-growth bug in transaction execution; this was a pure floating-point arithmetic bug in the reward-distribution sysvar calculation, in a completely different code path, one epoch later.

Root cause¶

Solana's epoch-boundary inflation calculation raises a tapering factor to a fractional power (roughly initial_rate × (1 − taper)^year) once per epoch to compute the total lamports available for that epoch's staking rewards. The validator's implementation used its language runtime's general-purpose floating-point power function for this. That function is not required to be correctly rounded — it is permitted to differ from a mathematically exact result by a small, implementation-defined margin, and different math libraries make different trade-offs here. The reference clients this project targets both resolve their power function to the platform C library's pow(), which — for this specific input — produces a floating-point result exactly one ULP above what the validator's own runtime function produced.

That one-ULP difference in the reward rate survives the subsequent multiplication and truncation into an integer lamport amount as a difference of exactly one lamport in the total reward pool for the epoch — verified by hand-reproducing both code paths' arithmetic bit-for-bit outside the validator and confirming the exact divergence point. Because that reward-pool total is written into an on-chain sysvar that every account touched by that epoch's reward distribution derives from, the one-lamport difference propagated into the accounts lattice hash and, from there, into the bank hash — the validator's root-advancement safety check correctly refused to advance past the divergent slot rather than silently accepting it.

The fix¶

Replaced the general-purpose power function in the inflation-reward calculation with an implementation byte-identical to the reference C math library on this input class — eliminating the ULP gap at its source rather than working around its symptom.
Added a permanent known-answer test that pins the exact epoch's reward-pool value (bit-for-bit) as a regression gate.
Audited the rest of the epoch/reward/stake-warmup code path for other uses of the same general-purpose transcendental math functions, to check for the same class of rounding gap elsewhere.

Proof¶

Deterministic offline reproduction. The pre-fix code reproduces the incorrect reward-pool value and the incorrect bank hash bit-for-bit from a captured pre-boundary state; the fixed code reproduces the canonical bank hash for the same slot range exactly, whole-bank (not just the isolated arithmetic).
No regression. The general offline golden-replay range remains byte-identical to canonical with the fix applied.

This deterministic reproduction is now kept as a permanent, named regression gate — see the gate battery.

Status and residual risk¶

Resolved and deployed. The fix has been live since shortly after the incident and cleared all gates above. The specific mechanism (a not-necessarily-correctly-rounded general-purpose power function used in a consensus-critical calculation) is closed for this call site; the same audit is a standing practice for any new transcendental math introduced on a consensus path going forward.

Lessons¶

"Close enough" floating point is not a category that exists in a hash-based consensus system. A general-purpose math function that is allowed to be off by one ULP is a latent consensus bug the moment its output feeds a hash, even though one ULP is completely invisible in almost every other context.
A stale diagnostic constant can look like a root cause. The early, misleading signal here came from the node's own logging, not from the actual defect — a reminder to treat log-line "expected" values as sanity checks, not ground truth, and to re-derive expected values independently rather than trusting a comment that may predate the current network parameters.
PoH-identity is a fast, cheap discriminator. Confirming the node's Proof-of-History hash matched the canonical blockhash immediately ruled out an entire class of hypotheses (shred/network/replication) and focused the investigation on execution — the same technique used in the truncated-block postmortem to rule out equivocation.