Skip to content

Observability

Docs-site note (2026-07-11): Part of the planned Vexor documentation site (Anza-style). Every claim on this page is grounded in Vexor's metrics reporter source (core/metrics_reporter.zig) — the measurement names, field names, cadence, and failure-handling described below are read directly from that file, not inferred.

Overview

Vexor ships a metrics reporter that submits validator telemetry to the standard Solana metrics cluster (metrics.solana.com) using the same wire format and configuration convention Agave uses. Point it at the same InfluxDB endpoint and the standard Anza-style dashboards recognize and plot a Vexor node exactly like any other client — no custom tooling required.

On top of that Agave-compatible baseline, Vexor emits a second layer of its own measurements that expose internals Agave's schema has no room for: replay throughput, vote-census outcomes, TVU shred handling, AF_XDP kernel counters, process resource usage, and per-mount disk usage.

Before you start

  • Metrics are optional. A node with no SOLANA_METRICS_CONFIG set, or an invalid one, runs and votes identically to one reporting cleanly — there is no metrics-config requirement to participate in consensus.
  • This is testnet. Vexor is a testnet-only, 0.9.x pre-production client. If you're pointing at the public metrics endpoint, use testnet credentials for the cluster you're actually joining — see Getting credentials below.
  • The never-crash contract is designed and code-audited, not yet field-proven. The spawn-path hardening described in The never-crash contract was landed the same day as this page (2026-07-11), and has not yet completed a live boot-smoke validation pass. Treat the guarantee as verified-in-isolation until that pass lands.
  • You don't need anything Vexor-specific to start. The baseline Agave-compatible layer (validator-new, cpu-stats, memory-stats, and so on) needs nothing beyond the one environment variable in Configuration. The vexor-* layer in The Vexor rich layer rides on the same connection automatically — there's no separate switch to flip.

Configuration — SOLANA_METRICS_CONFIG

Metrics are configured with the same environment variable Agave reads, in the same format:

SOLANA_METRICS_CONFIG="host=<url>,db=<db>,u=<user>,p=<password>"
  • All four keys (host, db, u, p) are required; any order is accepted.
  • Parsing mirrors Agave's get_metrics_config: the string is split on ,, then each pair is split on = into exactly two parts. A pair that doesn't split cleanly, an unrecognized key, or a missing key disables metrics — it never crashes the validator (see The never-crash contract).
  • If the variable is unset, the reporter does not start at all — this is the master on/off switch.

Getting credentials

For the current public testnet metrics write credentials, see docs.anza.xyz/clusters/available — Vexor consumes them exactly as published there, no translation needed. Set the variable wherever the rest of your deploy environment is set; see Environment Variable Reference and Deploying Vexor for how Vexor's own reference deployment wires it in.

The password is never logged. Any log line that echoes the config redacts it to p=****, and it is never baked into the binary — it only ever exists as an environment value on the deploy command line.

How the write happens

Vexor builds the same write URL Agave builds:

{host}/write?db={db}&u={user}&p={password}&precision=n

Credentials travel in the query string (not HTTP basic auth) — matching Agave's InfluxDbMetricsWriter. Points are POSTed over a single persistent libcurl handle where libcurl is available, with a subprocess curl call as a fallback if it isn't.

Wire format

Every point is emitted in standard InfluxDB line protocol, byte-shaped the way Agave's serialize_points emits it:

measurement,host_id=<id>[,tag=value...] field1=value1[,field2=value2...] <timestamp_ns>
  • Integer fields carry an i suffix (shreds_received=1042i), floats are plain decimals, strings are double-quoted with embedded " escaped as \", and booleans are true/false.
  • host_id — the validator's identity pubkey, base58-encoded — is always the first tag on every point, matching Agave's set_host_id convention. This is what lets a dashboard filter to a single node.

Agave-compatible measurements

These are the same measurement names and field shapes Agave's system-monitor-service and core validator emit, so existing dashboards built against Agave nodes plot Vexor data without modification:

Measurement Emitted Fields
validator-new once, at boot id, version, cluster_type, elapsed_ms, waited_for_supermajority, shred_version
cpu-stats every flush cpu_num, cpu0_freq_mhz, average_load_one_minute, average_load_five_minutes, average_load_fifteen_minutes, total_num_threads
memory-stats every flush total, swap_total, buffers_bytes, cached_bytes, free_percent, used_bytes, avail_percent, buffers_percent, cached_percent, swap_free_percent
net-stats-validator from the 2nd flush on per-interval deltas of UDP (/proc/net/snmp) and network-device (/proc/net/dev) counters — in/out datagrams, errors, drops, bytes, packets
disk-stats from the 2nd flush on per-interval deltas of /sys/block/*/stat counters — reads, writes, discards, flushes, time-in-IO, summed across physical disks (loop/dm/md devices excluded)
optimistic_slot every flush (when replay data is valid) slot
metrics every flush (bookkeeping) points_written, num_points, points_lost, points_buffered, secs_since_last_write

Net-stats and disk-stats need two samples to compute a delta, so they don't appear on the very first flush — this is expected, not a fault.

optimistic_slot honesty note: Agave derives this from a dedicated optimistic-confirmation vote tracker. Vexor does not currently run a separate optimistic-confirmation stream, so it reports the root slot instead — a conservative lower bound (a rooted slot is, by definition, already optimistically confirmed). The dashboard line is never ahead of reality; it may lag slightly behind where a full optimistic-confirmation tracker would put it.

The Vexor rich layer

Beyond the Agave-compatible baseline, Vexor emits its own vexor-* measurements that expose subsystems Agave's schema doesn't have a slot for. These are additive — nothing here changes or replaces an Agave-compatible point.

Measurement Fields
vexor-replay shreds_received, invalid_shreds, slots_replayed, successful_txs, failed_txs, votes_sent, blocks_produced, slot_queue_drops, root_slot
vexor-vote-census eligible, cast, fallback_decided, fallback_cast, silent_withhold — cumulative-since-boot counters of how many votes the node was eligible to cast versus what actually landed
vexor-tvu shreds_received, shreds_inserted, shreds_duplicate, shreds_invalid, zc_version_rejects, repairs_sent, repairs_received, repairs_served, repair_requests_received, slots_completed, max_slot_seen, shreds_retransmitted, repairs_dropped_ratelimit, rx_shed
vexor-afxdp rx_dropped, rx_invalid_descs, tx_invalid_descs, rx_ring_full, rx_fill_ring_empty_descs, tx_ring_empty_descs — read directly from the kernel via getsockopt(SOL_XDP, XDP_STATISTICS); only emitted when AF_XDP is active
vexor-process rss_bytes, vsize_bytes, utime_ticks, stime_ticks, num_threads — from /proc/self/stat
vexor-disk-usage total_bytes, avail_bytes, used_percent, tagged by mount_role (ledger, accounts, snapshots) — one point per configured mount

All of these are populated from existing validator counters (atomic loads via a callback supplied at startup) or read-only system calls — the reporter never adds instrumentation to a hot path to feed itself.

vexor-afxdp needs AF_XDP zero-copy networking active (VEX_ENABLE_AFXDP=1) to emit — on a node running the standard AF_INET path it simply doesn't appear, same as net-stats-validator needing a second sample.

Cadence

The reporter flushes once every 10 seconds (Agave's MetricsAgent write frequency), with a 5-second HTTP timeout per submission. If a submission fails, the batch is dropped — never queued and never retried — so a metrics outage cannot turn into a retry storm or unbounded memory growth on the validator. A log line notes when submission starts failing and again when it recovers, so an operator can see metrics-endpoint health without the validator's own operation being affected either way.

The never-crash contract

The reporter runs on a single dedicated background thread (self-niced to the lowest priority, never pinned to a consensus core) and is designed so that no failure in the metrics path can take the validator down. This is the intended contract and the source is written to guarantee it; note that the spawn-path hardening below was landed same-day (2026-07-11) and, as of this writing, has not yet completed its live boot-smoke validation pass — treat it as designed-and-tested-in-isolation rather than field-proven:

  • Config guard. Parsing SOLANA_METRICS_CONFIG never panics on any input, including truncated, oversized, or malformed values — a bad config just disables metrics and logs why.
  • Offline guard. The reporter refuses to start at all under VEX_LEDGER_REPLAY or VEX_SNAPSHOT_OFFLINE, so offline replay and golden-vector runs stay network-silent by construction.
  • Startup guard. Every allocation and thread-spawn step on the init path is wrapped so a failure (out of memory, thread-limit exhaustion, etc.) logs a warning and leaves metrics disabled, rather than propagating.
  • Runtime guard. A failed HTTP submission drops that batch and moves on; it is not treated as a fatal condition anywhere in the reporter.

The design principle is simple: metrics are strictly additive telemetry. A validator with metrics fully broken or fully absent behaves identically, on the consensus path, to one with metrics healthy.

Status

The metrics reporter is part of Vexor's core binary. The Agave-compatible measurement set lets standard dashboards recognize a Vexor node out of the box; the vexor-* layer is what operators reach for when they need visibility into Vexor-specific internals (replay throughput, vote census, TVU shred handling, AF_XDP) that no Agave dashboard was built to show.

See also

  • Environment Variable Reference — where SOLANA_METRICS_CONFIG fits alongside Vexor's other operator-facing environment variables.
  • Deploying Vexor — the deploy contract this reporter's startup sits inside.
  • Hardware — prerequisites for the vexor-afxdp measurement set.