Skip to content

Observability

Docs-site note (2026-07-11): Part of the planned Vexor documentation site (Anza-style). Every claim on this page is grounded in Vexor's metrics reporter source (core/metrics_reporter.zig) — the measurement names, field names, cadence, and failure-handling described below are read directly from that file, not inferred.

Overview

Vexor ships a metrics reporter that submits validator telemetry to the standard Solana metrics cluster (metrics.solana.com) using the standard Solana validator metrics wire format and configuration convention. Point it at the same InfluxDB endpoint and the standard dashboards used across the ecosystem recognize and plot a Vexor node exactly like any other client — no custom tooling required.

On top of that standard baseline, Vexor emits a second layer of its own measurements that expose internals the standard schema has no room for: replay throughput, vote-census outcomes, TVU shred handling, AF_XDP kernel counters, process resource usage, and per-mount disk usage.

Before you start

  • Metrics are optional. A node with no SOLANA_METRICS_CONFIG set, or an invalid one, runs and votes identically to one reporting cleanly — there is no metrics-config requirement to participate in consensus.
  • This is testnet. Vexor is a testnet-only, 0.9.x pre-production client. If you're pointing at the public metrics endpoint, use testnet credentials for the cluster you're actually joining — see Getting credentials below.
  • The never-crash contract is designed and code-audited, not yet field-proven. The spawn-path hardening described in The never-crash contract was landed the same day as this page (2026-07-11), and has not yet completed a live boot-smoke validation pass. Treat the guarantee as verified-in-isolation until that pass lands.
  • You don't need anything Vexor-specific to start. The baseline standard metrics layer (validator-new, cpu-stats, memory-stats, and so on) needs nothing beyond the one environment variable in Configuration. The vexor-* layer in The Vexor rich layer rides on the same connection automatically — there's no separate switch to flip.

Configuration — SOLANA_METRICS_CONFIG

Metrics are configured with the standard SOLANA_METRICS_CONFIG environment variable, in the standard format:

SOLANA_METRICS_CONFIG="host=<url>,db=<db>,u=<user>,p=<password>"
  • All four keys (host, db, u, p) are required; any order is accepted.
  • Parsing follows the standard SOLANA_METRICS_CONFIG convention: the string is split on ,, then each pair is split on = into exactly two parts. A pair that doesn't split cleanly, an unrecognized key, or a missing key disables metrics — it never crashes the validator (see The never-crash contract).
  • If the variable is unset, the reporter does not start at all — this is the master on/off switch.

Getting credentials

For the current public testnet metrics write credentials, see docs.anza.xyz/clusters/available — Vexor consumes them exactly as published there, no translation needed. Set the variable wherever the rest of your deploy environment is set; see Environment Variable Reference and Deploying Vexor for how Vexor's own reference deployment wires it in.

The password is never logged. Any log line that echoes the config redacts it to p=****, and it is never baked into the binary — it only ever exists as an environment value on the deploy command line.

How the write happens

Vexor builds the standard metrics write URL:

{host}/write?db={db}&u={user}&p={password}&precision=n

Credentials travel in the query string (not HTTP basic auth), per the standard metrics-write convention. Points are POSTed over a single persistent libcurl handle where libcurl is available, with a subprocess curl call as a fallback if it isn't.

Wire format

Every point is emitted in standard InfluxDB line protocol, byte-shaped per the standard Solana validator metrics convention:

measurement,host_id=<id>[,tag=value...] field1=value1[,field2=value2...] <timestamp_ns>
  • Integer fields carry an i suffix (shreds_received=1042i), floats are plain decimals, strings are double-quoted with embedded " escaped as \", and booleans are true/false.
  • host_id — the validator's identity pubkey, base58-encoded — is always the first tag on every point, per the standard convention. This is what lets a dashboard filter to a single node.

Standard measurements

These are the same measurement names and field shapes used across the Solana validator ecosystem's standard telemetry, so existing dashboards plot Vexor data without modification:

Measurement Emitted Fields
validator-new once, at boot id, version, cluster_type, elapsed_ms, waited_for_supermajority, shred_version
cpu-stats every flush cpu_num, cpu0_freq_mhz, average_load_one_minute, average_load_five_minutes, average_load_fifteen_minutes, total_num_threads
memory-stats every flush total, swap_total, buffers_bytes, cached_bytes, free_percent, used_bytes, avail_percent, buffers_percent, cached_percent, swap_free_percent
net-stats-validator from the 2nd flush on per-interval deltas of UDP (/proc/net/snmp) and network-device (/proc/net/dev) counters — in/out datagrams, errors, drops, bytes, packets
disk-stats from the 2nd flush on per-interval deltas of /sys/block/*/stat counters — reads, writes, discards, flushes, time-in-IO, summed across physical disks (loop/dm/md devices excluded)
optimistic_slot every flush (when replay data is valid) slot
metrics every flush (bookkeeping) points_written, num_points, points_lost, points_buffered, secs_since_last_write

Net-stats and disk-stats need two samples to compute a delta, so they don't appear on the very first flush — this is expected, not a fault.

optimistic_slot honesty note: this measurement is typically derived from a dedicated optimistic-confirmation vote tracker. Vexor does not currently run a separate optimistic-confirmation stream, so it reports the root slot instead — a conservative lower bound (a rooted slot is, by definition, already optimistically confirmed). The dashboard line is never ahead of reality; it may lag slightly behind where a full optimistic-confirmation tracker would put it.

The Vexor rich layer

Beyond the standard baseline, Vexor emits its own vexor-* measurements that expose subsystems the standard schema doesn't have a slot for. These are additive — nothing here changes or replaces a standard point.

Measurement Fields
vexor-replay shreds_received, invalid_shreds, slots_replayed, successful_txs, failed_txs, votes_sent, blocks_produced, slot_queue_drops, root_slot
vexor-vote-census eligible, cast, fallback_decided, fallback_cast, silent_withhold — cumulative-since-boot counters of how many votes the node was eligible to cast versus what actually landed
vexor-tvu shreds_received, shreds_inserted, shreds_duplicate, shreds_invalid, zc_version_rejects, repairs_sent, repairs_received, repairs_served, repair_requests_received, slots_completed, max_slot_seen, shreds_retransmitted, repairs_dropped_ratelimit, rx_shed
vexor-afxdp rx_dropped, rx_invalid_descs, tx_invalid_descs, rx_ring_full, rx_fill_ring_empty_descs, tx_ring_empty_descs — read directly from the kernel via getsockopt(SOL_XDP, XDP_STATISTICS); only emitted when AF_XDP is active
vexor-process rss_bytes, vsize_bytes, utime_ticks, stime_ticks, num_threads — from /proc/self/stat
vexor-disk-usage total_bytes, avail_bytes, used_percent, tagged by mount_role (ledger, accounts, snapshots) — one point per configured mount

All of these are populated from existing validator counters (atomic loads via a callback supplied at startup) or read-only system calls — the reporter never adds instrumentation to a hot path to feed itself.

vexor-afxdp needs AF_XDP zero-copy networking active (VEX_ENABLE_AFXDP=1) to emit — on a node running the standard AF_INET path it simply doesn't appear, same as net-stats-validator needing a second sample.

Cadence

The reporter flushes once every 10 seconds, matching the standard metrics write frequency, with a 5-second HTTP timeout per submission. If a submission fails, the batch is dropped — never queued and never retried — so a metrics outage cannot turn into a retry storm or unbounded memory growth on the validator. A log line notes when submission starts failing and again when it recovers, so an operator can see metrics-endpoint health without the validator's own operation being affected either way.

The never-crash contract

The reporter runs on a single dedicated background thread (self-niced to the lowest priority, never pinned to a consensus core) and is designed so that no failure in the metrics path can take the validator down. This is the intended contract and the source is written to guarantee it; note that the spawn-path hardening below was landed same-day (2026-07-11) and, as of this writing, has not yet completed its live boot-smoke validation pass — treat it as designed-and-tested-in-isolation rather than field-proven:

  • Config guard. Parsing SOLANA_METRICS_CONFIG never panics on any input, including truncated, oversized, or malformed values — a bad config just disables metrics and logs why.
  • Offline guard. The reporter refuses to start at all under VEX_LEDGER_REPLAY or VEX_SNAPSHOT_OFFLINE, so offline replay and golden-vector runs stay network-silent by construction.
  • Startup guard. Every allocation and thread-spawn step on the init path is wrapped so a failure (out of memory, thread-limit exhaustion, etc.) logs a warning and leaves metrics disabled, rather than propagating.
  • Runtime guard. A failed HTTP submission drops that batch and moves on; it is not treated as a fatal condition anywhere in the reporter.

The design principle is simple: metrics are strictly additive telemetry. A validator with metrics fully broken or fully absent behaves identically, on the consensus path, to one with metrics healthy.

Status

The metrics reporter is part of Vexor's core binary. The standard measurement set lets existing dashboards recognize a Vexor node out of the box; the vexor-* layer is what operators reach for when they need visibility into Vexor-specific internals (replay throughput, vote census, TVU shred handling, AF_XDP) that the standard schema wasn't built to show.

See also

  • Environment Variable Reference — where SOLANA_METRICS_CONFIG fits alongside Vexor's other operator-facing environment variables.
  • Deploying Vexor — the deploy contract this reporter's startup sits inside.
  • Hardware — prerequisites for the vexor-afxdp measurement set.