Skip to content

Networking

Docs-site note (2026-07-11): Part of the Vexor documentation site (Anza-style). This page describes Vexor's network stack — how packets get from the wire into replay, and how the validator participates in shred propagation, repair, and gossip. Claims are grounded in the live source tree; where a path is gated or partial, it says so.

Overview

A Solana validator lives and dies on the network. It must ingest a firehose of shreds (the ~1228-byte, erasure-coded fragments a block is split into) at line rate, recover any that are lost, repair the rest from peers, discover and stay in sync with the cluster over gossip, and — as leader — broadcast its own block down a stake-weighted tree. Vexor's network stack is a clean-room Zig implementation of all of this, validated byte-for-byte against the Agave reference implementation (currently anchored to the newest testnet release, v4.2.0-beta.0 — the wire/behavior target, re-anchored as new releases ship), with Firedancer used as the engineering reference for the shred/FEC paths and design discipline (RX-only AF_XDP, no-LSM storage) and Sig consulted as an additional engineering reference where relevant.

The stack splits cleanly by transport, matching the cluster's own design:

Traffic Transport Why
Shreds (Turbine, repair responses) UDP — zero-copy receive via AF_XDP Tiny, erasure-coded packets; QUIC's per-stream overhead would only hurt
Transactions + TPU votes QUIC (TLS 1.3) Connection-oriented, flow-controlled, stake-prioritized ingest
Gossip UDP (CRDS / bincode) Bloom-filter pull/push dissemination

The network tiles are pinned: recv (shred ingest) on core 5, quic (QUIC pump) on core 6, the verify pool on cores 8–15, gossip on core 24, and repair on core 30. See the Architecture Overview for the full table.

AF_XDP zero-copy ingest

The shred firehose is the highest-rate inbound path, so it gets the fastest ingest mechanism Linux offers: AF_XDP. Instead of every packet traversing the kernel network stack and being copied into a socket buffer, AF_XDP delivers packets from the NIC straight into userspace ring buffers (UMEM) shared with the kernel — no per-packet copy, no stack traversal.

The four AF_XDP rings:

  • Fill ring (user → kernel): empty frame addresses the NIC may fill.
  • RX ring (kernel → user): descriptors of received packets.
  • TX ring (user → kernel): packets to send.
  • Completion ring (kernel → user): TX frames the NIC has finished sending.

Vexor binds AF_XDP in driver (zero-copy) mode on the Mellanox ConnectX-6 Dx (mlx5), whose driver registers ndo_bpf and supports the real --xdp-zero-copy bind. The RX ring is sized to the card's hardware maximum (8192, per ethtool -g). This RX path is what lets the recv tile keep up with Turbine at line rate.

Ingest backend selection is automatic and degrades gracefully:

Backend Throughput (order of magnitude) When
AF_XDP (kernel bypass) ~10M pps mlx5/ice driver + AF_XDP enabled
io_uring batch UDP ~3M pps fallback
standard UDP sockets ~1M pps universal fallback

Scope (do not over-read this): AF_XDP zero-copy is receive-only, by design — this matches Firedancer's own approach. Outbound packets (votes, repair requests, broadcast) go via standard kernel UDP send paths, not an AF_XDP TX ring. RX zero-copy is the win that matters: it is the ingest path that has to keep up with the Turbine firehose at line rate, where TX volume is a small fraction of RX and standard sends are not the bottleneck.

AF_XDP is enabled at deploy with VEX_ENABLE_AFXDP=1. See Network setup for NIC binding, queue/IRQ steering, and the cpuset, and Performance for the throughput tuning that pairs RX queues to the pinned tiles.

QUIC — transactions and TPU votes

Transactions and TPU votes use QUIC (over TLS 1.3), the same transport modern Solana clients use for the TPU. Vexor has a native Zig QUIC implementation — frame parsing, stream multiplexing, flow control, connection handling — not an FFI binding.

  • Ingest. The quic tile (core 6) terminates inbound QUIC connections and decodes the bincode-serialized transactions from QUIC streams. Solana's TPU limits apply (per-IP connection caps, per-connection stream caps, stake-weighted rate limiting).
  • TPU votes. Vexor can submit its votes natively over QUIC to the current leader's TPU. The QUIC client correctly negotiates the leader's uni-stream flow-control credit (leaders omit the initial max_streams_uni transport parameter and grant credit via MAX_STREAMS_UNI frames, which the byte-exact frame parser handles). A UDP relay fallback for votes exists alongside the QUIC path, so vote delivery is robust regardless of which submission path is active.

Shreds do not use QUIC — see the transport table above. QUIC is for the connection-oriented transaction/vote path only.

Turbine — shred propagation

Turbine is Solana's block-propagation protocol: the leader does not unicast its block to all 1000+ validators; it shreds the block and broadcasts down a stake-weighted tree, where each node forwards to a fan-out of children, so propagation is logarithmic in cluster size.

Vexor's Turbine tree (turbine_tree.zig) is a faithful port of Agave's get_nodes / cluster_nodes.rs: the candidate set is [self] ++ gossip TVU peers ++ all staked nodes, each weighted by its real epoch stake, sorted descending by (stake, pubkey). For each shred, a deterministic weighted shuffle (a Fenwick-tree shuffle seeded by (slot, shred index, shred type, leader pubkey), driven by a ChaCha8/20 RNG) selects the broadcast root and the children. The weighted-shuffle RNG is byte-exact to Agave's (weighted_shuffle.zig), down to the rand_chacha 0.9.0 block semantics — verified against Agave's hard-coded test vectors — because the chosen root must match what the rest of the cluster expects, or the shreds never fan out.

The real-stakes broadcast tree (block production)

The broadcast tree's correctness depends entirely on the stakes fed into it. A bug here was the root cause of produced blocks being skipped: with a uniform stake over gossip-only peers, the weighted shuffle picks a near-random, non-canonical broadcast root, the cluster's retransmit tree never receives the shreds at the expected root, the FEC sets never complete cluster-side, and the block is dropped.

The fix is the real-stakes path: feed the full epoch staked-node set with real stakes from the leader cache (the same map the stake-weighted repair path uses), driven by VEX_TURBINE_REAL_STAKES together with VEX_LEADER_BROADCAST (the canonical root only matters when the node is producing blocks). Both are now baked on by default at boot — an operator does not need to set either env explicitly; an explicit =0 overrides the bake and disarms it. The path auto-falls-back to uniform (with a loud warning) if the epoch stakes aren't populated yet. With real stakes on, leader windows have been accepted by the cluster (proven 4/4 at an epoch-977 window).

Turbine retransmit (forwarding received shreds to one's own tree children) has logic in source gated by VEX_TURBINE_RETRANSMIT, but it is dead code in every build: the -Dturbine_retransmit comptime option it depends on is hardcoded false in build.zig, with no CLI flag exposed to turn it on — so the env var can never take effect. Retransmit completeness remains an open item, not an operator-flippable feature today.

Repair — filling the holes

Turbine is best-effort; shreds get dropped. Repair is the recovery channel: when a slot is missing shreds, Vexor requests them directly from peers over UDP. The request types mirror Solana's:

Request Asks for
WindowIndex the shred at a specific (slot, index)
HighestWindowIndex the highest shred index known for a slot
Orphan the parent shred of an orphaned slot
AncestorHashes the hash chain for fork verification

The load-bearing decision in repair is which peer to ask. Vexor's targeting (repair_targeting.zig) is a pure, KAT-tested function extracted from the receive loop. It fixes a real convergence bug: the old "one missing shred → exactly one peer" round-robin would request a singleton hole from the same non-holder forever (a WindowIndex request gets no negative ack when a peer lacks the shred), which kept the node delinquent. The targeting function spreads requests so holes converge.

Repair peer selection can also be stake-weighted (-Drepair_stake_weighting, part of the canonical production build flag set): higher flow goes to advertisers and high-stake nodes first, which are more likely to have the shred and to answer promptly. Repair is consensus-neutral — it only governs which peer a missing shred is requested from; a repaired shred is still admitted only through the same verified ingest path (signature + merkle-root checks), so repair affects liveness/convergence, never what is accepted into a block.

Gossip — CRDS

Gossip is how Vexor discovers the cluster and stays in sync with cluster-wide soft state. It implements Solana's CRDS (Cluster Replicated Data Store): a bincode wire format disseminated by a Bloom-filter-based pull/push protocol over UDP.

  • Message types: pull_request, pull_response, push_message, prune_message, ping/pong.
  • CRDS values carried: ContactInfo (node addresses — gossip/TVU/repair/TPU ports + shred version), Vote records, EpochSlots (slot-completion bitmaps), and the other standard CRDS data types.
  • Discovery feeds Turbine and repair: the contact table is exactly the peer set the Turbine tree and the repair targeting draw from (filtered by matching shred version).

CRDS signature verification on ingest is implemented but gated (VEX_CRDS_VERIFY=off|log|reject, default off, not part of the baked-default set): log verifies and counts per-tag pass/fail while still ingesting (to observe the live fail-rate); reject drops only the invalid value. The canonical behavior (Agave verifies CRDS sigs on ingest and drops bad values) is the target; reject is enabled only after a live log window confirms the signable-bytes computation is byte-exact, so valid gossip is never false-rejected.

FEC — Reed-Solomon recovery and the dedup cache

Solana shreds are grouped into FEC sets and protected with Reed-Solomon erasure coding over the Galois field GF(2⁸) with primitive polynomial 0x11D (α = 2) — the same field Agave's reed-solomon-erasure and Firedancer's fd_reedsol use. As long as enough shreds in a set arrive (data or coding/parity), the missing ones can be reconstructed without a repair round-trip.

Recovery (receive path) — live. fec_resolver.zig (ported from Firedancer's fd_fec_resolver.c) recovers missing shreds from real cluster FEC sets using a Vandermonde matrix M = V·inv(top(V)) over that field — the same field and matrix the cluster's encoder uses, so the recovery is byte-correct. Merkle roots are reconstructed and verified per set (including SIMD-0340 chained merkle roots).

Encoding (produce path) — staged. The forward direction (data → parity shreds), needed for the leader to broadcast a recoverable block, applies the same field and matrix in the encode direction (shred_reedsol.zig). It is modular and gated behind -Dleader_mode at the call site — wired into the block-production path, not the always-on receive path.

The ed25519 FEC-set dedup cache

Every shred carries an ed25519 signature, and signature verification is the most expensive per-shred operation. But within one FEC set, all ~64 shreds share the same signature, leader pubkey, and merkle root. Vexor mirrors Agave's optimization (sigverify_shreds.rs): a bounded LRU keyed on (signature, pubkey, reconstructed merkle root). The first shred of a FEC set runs the full ed25519 verify; its ~63 siblings hit the cache and skip it.

This is a pure performance shortcut that never weakens verification: the leader pubkey is in the key (a slot-rewrite by a different leader → cache miss), the merkle root is reconstructed from this shred's own leaf+proof and covers the slot field (a content rewrite → different root → cache miss), and a key is only inserted after a successful verify — so a cache hit means "this exact tuple was already ed25519-verified." It requires the -Dfec_dedup build flag and is armed by VEXOR_ED25519_FEC_DEDUP, which is now baked on by default at boot on a binary built with that flag (an explicit =0 overrides the bake and disarms it); proven bank-neutral and correct on live shreds.

Note: shred signature verification today is per-shred (the dedup cache reduces it to roughly one ed25519 per FEC set). Batched AVX-512 sigverify is a separate, not-yet-wired throughput item — Vexor does not yet batch the way Firedancer does.

How it fits together

   NIC (ConnectX-6 Dx)
        │
        ├── UDP/AF_XDP ──► recv tile (5) ──► verify pool (8–15) ──► FEC resolver ──► replay (16)
        │                                       │ ed25519 + dedup     │ Reed-Solomon
        │                                       │                     │ recovery + repair fill
        │                                       ▼                     ▼
        │                                  (drops/holes) ──────► repair tile (30) ──► peers
        │
        ├── UDP (CRDS) ──► gossip tile (24) ──► contact table ──► feeds Turbine tree + repair peers
        │
        └── QUIC ──► quic tile (6) ──► tx decode ──► replay;   votes out ◄── txsend (28) via QUIC/UDP relay

   (LEADER) replay ──► produce (20) ──► Reed-Solomon encode ──► Turbine stake-weighted broadcast ──► cluster

Gossip supplies the peer set; AF_XDP and QUIC supply the inbound bytes; sigverify + FEC + repair turn a lossy firehose into complete, verified blocks for replay; and Turbine + the encoder carry Vexor's own blocks back out when it is leader.

Status (2026-07-11)

Path State
AF_XDP zero-copy RX (Mellanox mlx5) Live (VEX_ENABLE_AFXDP=1)
AF_XDP TX Not used, by design — TX goes via standard kernel UDP send paths (matches Firedancer's approach)
Native QUIC ingest + TPU votes (UDP relay fallback) Live
Turbine tree + byte-exact weighted shuffle Live (receive/retransmit-tree build)
Turbine real-stakes broadcast Baked on by default (VEX_TURBINE_REAL_STAKES + VEX_LEADER_BROADCAST); proven block acceptance
Turbine retransmit (forward received shreds) Dead code — -Dturbine_retransmit hardcoded false in build.zig, no CLI flag exposed; VEX_TURBINE_RETRANSMIT is comptime-pruned in every build
Repair (WindowIndex/Highest/Orphan/Ancestor + targeting fix) Live
Stake-weighted repair selection In production build (-Drepair_stake_weighting)
Gossip / CRDS discovery Live
CRDS inbound signature verify Gated (VEX_CRDS_VERIFY=off\|log\|reject, default off)
FEC Reed-Solomon recovery (receive) Live
FEC Reed-Solomon encoder (produce) Staged (-Dleader_mode)
ed25519 FEC-set dedup cache Baked on by default (-Dfec_dedup build flag + VEXOR_ED25519_FEC_DEDUP)
Batched AVX-512 sigverify Not yet wired (per-shred today)

For NIC binding, queue/IRQ steering, the cpuset, and AF_XDP enablement, see Network setup. For throughput tuning and how RX queues pair to the pinned tiles, see Performance. For the full baked-default env table, see Environment reference.