Troubleshooting¶

Docs-site note (2026-07-11): Part of the Vexor documentation site (Anza-style). This page is task-based — "my validator is doing X, here's what to check" — and every symptom→cause mapping below is grounded in the deploy pipeline's own self-checks (Deploy), the build's sanity checks (Build), the baked environment defaults (Environment Variable Reference), and VexLedger's documented behavior — not invented scenarios. Where this page doesn't have a proven answer yet, it says so instead of guessing.

Prerequisites and honest limits of this page¶

This page assumes you deployed with the deploy contract (stop → verify → capabilities → clean state → tune → launch → verify) — if you're launching the binary some other way, some of the log lines and guard checks referenced below won't apply.
Vexor is testnet-only, 0.9.x pre-production software. Some failure modes here are things we've actually hit and fixed; others (like a binary-to-binary VexLedger downgrade) simply haven't happened yet in practice, and this page says that plainly rather than inventing a recipe for them.
Vexor doesn't yet have a dedicated public support channel of its own. If you're stuck after working through this page, the fastest path is opening an issue against the project with the boot-log excerpt and cluster oracle output described below.

Check the cluster oracle before anything else¶

Before reading a single local log line, check what the public cluster thinks of your vote account. A validator's own RPC is not a reliable source of truth about the wider cluster — a minimal or partially implemented RPC surface can return stale or naive answers for cluster-wide facts like current epoch or leader schedule (see Deploy → Verifying the deploy is green). Query the public testnet RPC instead, keyed on your vote account pubkey:

curl -s https://api.testnet.solana.com -X POST -H 'content-type: application/json' \
  -d '{"jsonrpc":"2.0","id":1,"method":"getVoteAccounts",
       "params":[{"votePubkey":"<YOUR_VOTE_PUBKEY>"}]}' \
  | python3 -m json.tool

This one call tells you which of the sections below is actually relevant:

Vote account under delinquent, lastVote frozen → the process is likely wedged or crashed. Go to not catching up / delinquent.
Vote account under delinquent (or current but lastVote/rootSlot climbing slowly) right after a fresh boot → this may just be catch-up in progress, not a fault. Read the same section before restarting anything.
Everything green (current, lastVote and rootSlot both advancing each poll) → the validator is healthy; whatever you were chasing may be a metrics or logging question, not a consensus one — see where to find the logs.

Treat this as the capture-first step, not just a diagnostic: run it and save the output before you restart or wipe anything. A restart clears the exact evidence (process state, in-flight repair counters, the boot-log guard lines) that explains what actually happened.

My validator isn't catching up / shows delinquent¶

Work through these in order — each one is a real, previously-seen cause, not a guess.

1. Rule out "it's still catching up," not "it's broken"¶

A fresh boot legitimately takes time to reach the cluster tip, and an external process supervisor that restarts too eagerly during that window turns a slow-but-healthy boot into a restart loop that never finishes catching up — which looks exactly like an outage but is the supervisor causing one (see Deploy → Supervision and restart). If you run the validator under systemd or a similar supervisor with automatic restart:

# Check whether the process has been restarting repeatedly
systemctl status <your-vexor-unit> --no-pager -l | grep -i "restart\|active"

If it has, disarm automatic restart, let the current boot finish catching up (confirmed by lastVote and rootSlot both climbing in the oracle query above), and only then re-arm it.

2. Confirm the binary is actually a release build¶

A Debug binary is roughly 30× slower and will never keep up with replay and PoH hashing in real time, even though it starts and runs (see Build → Why --release=safe). Check the size:

ls -la zig-out/bin/vex-fd    # a correct --release=safe build is ~15-17 MB, not 50+ MB

The deploy pipeline's own binary-size sanity check refuses to launch anything far outside that range unless explicitly overridden — if you bypassed that check, this is the first thing to verify by hand.

3. Confirm the vote-program and BN254 consensus guards actually passed¶

The deploy pipeline hard-gates on two compiled-in markers before it will launch a binary at all (see Deploy → Consensus guards): the Vexor-authored vote-execution path (the voteforge marker — the vote program is a Vexor rewrite derived from Agave 4.2 semantics and is now the sole, unconditional live executor; a Sig-derived component served as the differential-test oracle during development and has been fully removed from the tree, Stage 8, 2026-07-12) and the BN254/Poseidon leaf-crypto path. You can check the same markers manually:

strings -n6 zig-out/bin/vex-fd | grep -c voteforge          # Vexor-authored vote-program guard present?
strings -n6 zig-out/bin/vex-fd | grep -c fd_bn254            # Firedancer Ballet BN254 linked?

voteforge should be nonzero — it is the only vote-execution path there is now. fd_bn254 should always read zero: crypto is unconditionally pure-Zig, and the Firedancer Ballet BN254 library it used to reference has been removed from the tree outright, so no current build ever links it. Because of that, every deploy now requires VEX_ALLOW_NO_BN254=1 set explicitly — without it, the deploy pipeline's BN254 guard aborts on seeing zero fd_bn254 refs (it can no longer tell "pure-Zig by design" apart from "Ballet accidentally not linked" any other way). This is not a transitional state; it is the guard's permanent behavior against any current binary. See the Environment Variable Reference. If the launch script's own boot log shows the voteforge guard printed as fail, the validator did not start with a safe binary — rebuild with the full -Dprod flag set (see Build → What -Dprod bundles) rather than a hand-typed flag list.

4. Check for a networking dead end: wrong shred version or unreachable entrypoints¶

If gossip shows zero valid peers, no shreds will ever arrive no matter how healthy the process is. The shred version is a cluster-generation fingerprint that changes across coordinated restarts — a stale value produces exactly this symptom (see Deploy → the CONFIG block). Confirm your configured shred version against a currently-healthy peer's gossip info, and confirm your entrypoints/known-validators list is current for the cluster you're joining.

5. Check whether a stale local snapshot is the starting point¶

If you're booting with VEX_REUSE_SNAPSHOT (skipping the fresh-download path) from a snapshot that's now far behind the cluster tip, the node has to replay a much longer distance to reach the tip than a fresh incremental snapshot would require. The safe default is VEX_FRESH_SNAPSHOT=1, which re-downloads a current base + incremental snapshot on every boot rather than trusting old local state (see Deploy → Snapshot policy). If the fresh download itself is failing (cluster snapshot infrastructure unreachable), fall back to the last known-good local snapshot rather than retrying in a loop.

6. Confirm you didn't disarm a catch-up-relevant baked default¶

Vexor bakes in-flight repair tracking and adaptive repair re-fire behavior on by default (VEX_REPAIR_INFLIGHT, VEX_CATCHUP_ADAPTIVE_REFIRE — see the baked necessity set). These only turn off if you (or your supervisor's environment) explicitly set them to 0. If your deploy command or unit file sets either to 0 deliberately for a diagnostic, that's a plausible reason catch-up is slower than expected — remove the override for a normal deploy.

VexLedger (blockstore) won't open, or fails after switching binaries¶

Anza's equivalent page here is about RocksDB column families and the ldb tool. None of that applies to Vexor — VexLedger is Vexor's own append-segment store, not a RocksDB database, so agave-ledger-tool/ldb cannot open its .seg files directly, exactly as neither can open Firedancer's fd_blockstore (see VexLedger → Cross-client fidelity). If you're looking for an ldb drop_column_family-style recipe, it doesn't exist for Vexor and isn't the right mental model — the compatibility guarantee is behavioral and at the RPC/wire boundary, not on-disk-format compatibility with RocksDB tooling.

What is documented and grounded:

Ordinary crash recovery is automatic. On boot, VexLedger reads its manifest, rebuilds the in-memory index from the surviving sealed segments, and replays the active segment's tail, truncating any torn final record from an unclean shutdown. This is deterministic and bounded — you shouldn't need to intervene by hand for a normal power-loss or kill -9 scenario (see VexLedger → Reads, writes, pruning, recovery).
A missing or empty ledger directory is not an error. If VEX_LEDGER is at its baked default (1) and the target directory is empty or absent, VexLedger initializes fresh state there — there's no equivalent of Agave's "predates the new column family" downgrade failure mode to plan around, because there's no RocksDB column-family schema underneath.
Downgrading to an older Vexor binary against an existing ledger directory has not been validated in production the way an upgrade has. Rather than inventing a compatibility claim we haven't proven, the honest and safe move today is to run Deploy step 4, clean state — wipe the working ledger and account cache and boot fresh — when moving to a binary from a meaningfully earlier point in the tree, instead of assuming the on-disk segment format is interchangeable across an unknown span of changes.

If the write path itself is failing (not a startup/open failure but errors while running), the far more common cause on a live node is the ledger's mount running out of space — see the next section.

Out of disk space¶

Vexor's reference deployment role-separates storage across three mounts specifically so this failure is easy to localize (see Hardware → Storage): OS + build tree, snapshots + working accounts DB, and the VexLedger blockstore each get their own drive. Check each independently rather than just root:

df -h / /mnt/snapshots /mnt/ledger    # substitute your own mount points

Ledger mount full. VexLedger bounds its own on-disk size by default — VEX_LEDGER_MAX_BYTES=107374182400 (100 GiB) is baked on unless you set it explicitly, with VEX_LEDGER_KEEP_SLOTS=216000 as the alternate slot-count policy (the byte cap wins if both are set — see the baked necessity set). Pruning is O(1) unlink() of whole sealed segments, so if the ledger mount is still filling up despite these caps, check whether your deploy command or supervisor unit is overriding either variable to a larger value (or to something that disables the cap) rather than assuming pruning isn't working.
Snapshots/accounts mount full. Every fresh boot with VEX_FRESH_SNAPSHOT=1 downloads a new base + incremental snapshot (see Deploy → Snapshot policy); old snapshot archives left behind by repeated boots on the same box are a common source of this filling up faster than expected. Clear stale archives on that mount, not the live ledger or accounts state.
Root/OS drive full. This is usually the build tree, not the running validator — repeated zig build invocations leave zig-out and cache artifacts behind. Clean build artifacts you don't need, or move the build tree off the OS drive.

For ongoing visibility rather than reacting after the fact, the metrics reporter emits a vexor-disk-usage point per mount, tagged by role (ledger, accounts, snapshots) — see Observability → The Vexor rich layer — so a dashboard alert on used_percent per role catches this before it becomes a startup failure.

PoH / replay rate is slower than the cluster target¶

Work through these in the order they're most likely to be the cause:

Confirm it's actually a release build, not Debug — see step 2 above. A Debug binary is the single most common cause of this exact symptom and is roughly 30× slower.
Confirm the build's -Dcpu target actually matches the hardware. Getting -Dcpu wrong doesn't break consensus, but it can silently disable ISA extensions (AVX-512, sha_ni, vaes) that Vexor's Ballet cryptography and hashing lean on for throughput (see Build → The one command). The reference box uses -Dcpu=znver4 for its EPYC 9374F; use the target that matches your CPU, or -Dcpu=native to let Zig detect the host.
Confirm the pinned tiles actually landed on the cores you intended. Vexor's replay, verify, and produce tiles are meant to run on dedicated, cache-warm cores (see Hardware → How Vexor maps onto the cores); the deploy pipeline's own final step confirms threads landed on their pinned cores and, if AF_XDP is enabled, that the zero-copy socket bound (see Deploy → step 7). If VEX_TASKSET_CORES or VEX_PARALLEL_EXEC_CORES don't match your actual core layout, hot tiles can end up sharing a core with background work, or wave-barrier parallel execution can have fewer workers than intended.
Check CPU frequency scaling, the same generic Linux tuning any latency-sensitive workload needs — this isn't Vexor-specific, but a validator whose cores are downclocked will show exactly this symptom:

lscpu | grep MHz

echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

# Example if your CPU's maximum is 4.3 GHz
echo 4300000 | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_min_freq

For ongoing visibility, the vexor-replay metrics measurement reports slots_replayed and slot_queue_drops per flush (see Observability → The Vexor rich layer) — a growing slot_queue_drops count is a direct signal that replay isn't keeping up, independent of what getVoteAccounts shows at any single poll.

Where to find the logs¶

Two layers, in the order to check them — cluster-wide truth first, then increasingly local detail:

The cluster oracle — described above. This is always the first check: it tells you whether you're chasing a real consensus/liveness problem or a local metrics/logging question.
The boot log. The launch script logs its own progress through the deploy contract's ordered steps, and the validator's own startup messages print a pass/fail line for each consensus guard (vote-program, BN254) plus, if AF_XDP is enabled, a final confirmation that the zero-copy socket bound and is receiving packets (see Deploy → Verifying the deploy is green). This is almost always the fastest place to find the actual failure reason for a boot that didn't come up at all.
Metrics. If the process is up but behaving oddly, vexor-replay, vexor-tvu, vexor-afxdp, and vexor-disk-usage (see Observability) expose the internal counters — shred drops, repair backlog, AF_XDP ring pressure, per-mount disk usage — that explain why, without needing to attach a debugger or add print statements.
Diagnostic environment variables, as a last resort for a genuine parity or crash investigation. These default off and aren't part of a normal deploy — account read/write recorders, state/field dumps, lt-hash verification, vote/gossip probes, SBPF VM tracing, and an allocator-debug mode are all listed in Environment Variable Reference → Diagnostic and forensic flags. Confirm the exact effect of one in source before relying on it — the same page notes that some historical variable names no longer do anything in the current binary.

Capture before you restart. If you're looking at something that resembles a wedge or a divergence rather than an ordinary slow catch-up, save the cluster-oracle output and the relevant boot-log lines before you restart the process or wipe state — a restart is exactly the action that destroys the evidence that would have explained what happened.

Symptom → likely cause quick reference¶

Symptom	Likely cause	Where to check
`delinquent` in `getVoteAccounts`, `lastVote` frozen	Process crashed or wedged	Boot log, process liveness
`delinquent`/slow, but `lastVote` and `rootSlot` are climbing	Still catching up after a fresh boot — not broken	Don't restart; see catch-up section
Auto-restart loop right after a deploy	Supervisor re-armed before CURRENT was confirmed	Deploy → Supervision and restart
Zero gossip peers / no shreds arriving	Stale shred version or dead entrypoints	Deploy → CONFIG block
Validator refuses to launch / aborts at startup	A consensus guard (vote-program or BN254) failed	Build → Verifying the flags landed
Binary is far larger than ~15–17 MB	Accidental Debug build	Build → Binary size as a sanity check
Ledger mount filling despite prune settings	`VEX_LEDGER_MAX_BYTES`/`VEX_LEDGER_KEEP_SLOTS` overridden or unset upstream	Out of disk space
VexLedger fails to open after a binary downgrade	Unvalidated on-disk compatibility across an older binary	VexLedger section
Slow PoH/replay, binary and guards all check out	Wrong `-Dcpu` target, core pinning mismatch, or CPU governor	PoH / replay rate section

Status¶

This page reflects the deploy pipeline's actual self-checks, the build's actual sanity checks, and VexLedger's documented recovery behavior as of 2026-07-11. Where a failure mode (like a binary-to-binary VexLedger downgrade) hasn't been exercised in practice, that's stated explicitly rather than filled in with an assumption.