Network Setup — AF_XDP Ingest and Core Pinning¶

Docs-site note (updated 2026-07-11): Part of the Vexor documentation site (Anza-style). This page documents how the live testnet validator ingests the network firehose and how its work is laid out across the CPU. Every value below is read directly from the deploy script (deploy.sh), the topology map in the binary (src/vex_topo.zig), and the live /proc thread affinities — not from memory. As of this update, most runtime feature envs are baked to their proven values at boot (see Environment Variable Reference); the deploy command below shows only the operator-facing, machine-specific settings you still need to pass.

This page is about the datapath and CPU layout: enabling AF_XDP zero-copy ingest on the Mellanox NIC, the NIC/queue/IRQ tuning that makes it work, the static tile→core pinning model, and the runtime verifiers that confirm both. For the hardware these run on see Hardware; for the protocol-level networking design see Architecture: Networking; for the kernel/sysctl knobs see Performance: Tuning.

Two ingest modes: kernel-UDP and AF_XDP zero-copy¶

Vexor can receive Turbine shreds two ways, selected at deploy time by the VEX_ENABLE_AFXDP environment variable:

Mode	`VEX_ENABLE_AFXDP`	Receive path	Active NIC port
Kernel-UDP (default fallback)	`0` (default)	Standard kernel UDP sockets, NAPI/GRO consumer	`enp129s0f0np0`
AF_XDP zero-copy (proven, recommended)	`1`	NIC DMAs shreds straight into a userspace ring (XSK), no per-packet kernel-stack copy	`enp129s0f1np1`

AF_XDP is opt-in — the default deploy is kernel-UDP, which is the proven-safe path on the management port. The live testnet validator runs with AF_XDP enabled (VEX_ENABLE_AFXDP=1), which is the recommended high-throughput configuration on the dedicated Mellanox ConnectX-6 Dx shred port. When AF_XDP is on, all NIC tuning, XDP attach, and the gratuitous-ARP move to the dedicated data port enp129s0f1np1; the management/SSH port enp129s0f0np0 is never touched.

VEX_BINARY=<pinned> VEX_ENABLE_AFXDP=1 VEX_FRESH_SNAPSHOT=1 VEX_PARALLEL_EXEC=1 \
VEX_PARALLEL_EXEC_CORES=25,26 VEX_TASKSET_CORES=<mask> bash deploy.sh dev

VEX_BINARY, VEX_PARALLEL_EXEC_CORES, and VEX_TASKSET_CORES are machine-specific and are never baked — you supply them for your own box. VEX_ENABLE_AFXDP, VEX_FRESH_SNAPSHOT, and VEX_PARALLEL_EXEC are shown explicitly here for clarity, but the last of the three is also one of the ~19 feature envs the binary bakes to its proven value on boot if you leave it unset (see Environment Variable Reference). With VEX_ENABLE_AFXDP=1, deploy.sh adds the launch flags --enable-af-xdp --xdp-zero-copy and routes its NIC ops to enp129s0f1np1. With it unset/0, neither flag is passed and the channel-reopen ethtool operations (which would reset the link) are skipped entirely.

Prerequisites for AF_XDP¶

AF_XDP needs capabilities, an unlocked-memory limit, and unprivileged-BPF enabled. deploy.sh sets all of these before launch:

Requirement	What / why	Set by
File capabilities	`cap_net_raw,cap_net_admin,cap_bpf,cap_perfmon+ep` on the binary — lets a non-root validator attach XDP and load the BPF program	`setcap` (Step 3)
`ulimit -l unlimited`	AF_XDP's UMEM (the shared frame pool) must be locked in RAM; a 32K-frame × 2048B socket needs ~64 MB locked, and headroom for scaling	`ulimit -l` (Step 5)
`kernel.unprivileged_bpf_disabled=0`	Debian defaults to `2` (disabled + locked), which makes `BPF_PROG_LOAD` return `EACCES` even with `CAP_BPF`	`sysctl` (Step 5)
`net.core.bpf_jit_enable=1`	JIT-compile the XDP program for line-rate redirect	`sysctl` (Step 5)

If any of these is missing, AF_XDP silently falls back to copy mode (or fails to bind) — the verifier in Confirming the datapath catches this.

NIC, queue, and IRQ tuning for AF_XDP¶

The Mellanox mlx5 driver needs specific settings for AF_XDP zero-copy to engage correctly. deploy.sh Step 5a/5b clears stale state, then applies these to the data port enp129s0f1np1:

Clean slate first¶

Before re-tuning, deploy.sh detaches any existing XDP program (ip link set dev … xdp off) and wipes Vexor's BPF pin directory (/sys/fs/bpf/vexor). Stale pins or a leftover XDP program cause rx_xsk_buff_alloc_err to storm and shreds to be dropped at the NIC→userspace boundary, so the nuke is idempotent and runs every deploy.

Queue layout — the XSK binds queue 0¶

The validator binds its AF_XDP socket (XSK) to queue 0. The tuning reserves q0 exclusively for that socket:

ethtool -L … combined 4 — open four combined RX queues.
ethtool -X … start 1 equal 3 — RSS indirection spreads all non-steered traffic across q1–q3 only, so the default hash never lands anything on q0.
ntuple flow steering routes Vexor's three UDP ports explicitly. In the default q0 steering mode, shred (8003), gossip (8001), and repair (8002) all go to q0; the XDP port_filter program on q0 redirects 8003 into the XSK and XDP_PASSes 8001/8002 straight to the kernel UDP sockets (Vexor keeps gossip + repair on kernel-UDP by design — only Turbine rides the XSK). A legacy "spread" mode (VEX_AFXDP_STEER_MODE=spread) puts gossip→q1, repair→q2; it left the XSK starved on this NIC and is kept only as a fallback.

Why queue 0 matters: an AF_XDP socket bound to q0 with traffic steered to a different queue leaves no XSK listener for shreds — repair can never fill missing FEC sets and the validator stalls at boot. Always confirm the XSK is on q0 (see Confirming the datapath).

mlx5 driver knobs (AF_XDP only)¶

Setting	Value	Why
`rx_striding_rq`	off	Striding RQ refills the fill-ring in all-or-nothing 64-frame batches → a low ring storms `rx_xsk_buff_alloc_err`. Cyclic RQ accepts partial allocation. (The #1 zero-copy fix.)
`rx_cqe_compress`	off	Compressed CQEs fail XSK striding validation.
ring size (`-G rx/tx`)	2048	8192 fails XSK validation ("RQ length too big for XSK").
adaptive-rx / adaptive-tx	off	Must be off before setting static `rx-usecs`, or the driver overrides it.
`rx-usecs` / `tx-usecs`	20	Static interrupt coalescing for predictable latency.
GRO / LRO / TSO / GSO	off	Shreds are single-datagram; coalescing hurts per-shred latency.
`rx-udp-gro-forwarding`	off	Vexor terminates UDP, it doesn't forward.
`napi_defer_hard_irqs` / `gro_flush_timeout`	0 under AF_XDP	These are a pair for a busy-poll NAPI consumer; the XSK is drained by Vexor's userspace tile, so deferral just delays RX. (Kernel-UDP keeps the proven `2`/`200000`.)

IRQ affinity (with the cores-0–4 wall-off)¶

When the optional cgroup-v2 cpuset wall-off is armed (VEX_CPUSET_WALLOFF=1), deploy.sh pins the shred-port q0 completion IRQ (mlx5_comp0) to core 4. Core 4 shares its CCX1 L3 slice with the receive tile on core 5, and is OS-reserved under the wall-off, so it is a legitimate kernel-IRQ home that lowers RX-completion latency for the zero-copy turbine path. The IRQ number is derived dynamically from the NIC's PCI address; the pin is a no-op if not found and is reversible.

The core-pinning / tiling model¶

Vexor runs as a set of pinned tiles — dedicated threads, each owning a specific core, Firedancer-style ("one tile per core, never reshuffle"). This is implemented statically in the binary (src/vex_topo.zig): each tile calls sched_setaffinity to pin itself to its assigned core as it spawns. The static map is the live model.

Live tiling table (EPYC 9374F, 32 cores)¶

Core(s)	Tile	Role
5	recv	AF_XDP TVU receive thread — dedicated, drains the XSK
6	quic-pump	QUIC ingest pump (may be dormant if gated)
8–15	verify (×8)	Ed25519 signature-verify workers (`8 + worker_id`)
16	replay	The hot replay loop (~95% CPU) — bank execution + PoH
20	produce	Block production (leader path; dormant unless leader)
24	gossip	Cluster gossip / CRDS
25, 26	wave	Parallel-exec WavePool workers (when `VEX_PARALLEL_EXEC=1`)
28	txsend	Vote / transaction send; native QUIC vote poller self-pins here
29	sysvar	Sysvar update tile
30	repair	Repair request/serve tile
31	watchdog	Liveness watchdog
4	ledger tile	VexLedger consumer — cold core, off the consensus path
1–3	dynamic relief	CCX0 reserve for bursty/floater work — kept off the hot tiles
0	—	Left to the OS / main thread

The EPYC 9374F's 8 CCX cache-complexes (4 cores each, 32 MiB L3 per CCX) make this layout cache-aware: consensus-critical tiles sit on cache-warm cores and never share a core with background work; the VexLedger consumer is fenced to a cold core (4) so persistence can never stall replay or voting; and dynamic relief / builds are fenced to CCX0 (cores 1–3) so a compile can't evict the hot working set.

`VEX_TASKSET_CORES` — the floater carve-out¶

The validator process is launched under taskset -c "$TASKSET_CORES". This mask applies only to unpinned / floater threads — the static tiles above override it in code via sched_setaffinity, so the mask is not what places the consensus tiles. Its job is to confine everything else to a safe window:

`VEX_ENABLE_AFXDP`	Default `TASKSET_CORES`	Note
`1` (AF_XDP)	`1-3,5-27`	CCX0 relief (1–3) included; core 4 excluded so the recv reservation holds
`0` (kernel-UDP)	`1-27`	No core-4 reservation needed

Two structural protections apply automatically:

Wave-core auto-carve: when VEX_PARALLEL_EXEC=1, the WavePool cores (VEX_PARALLEL_EXEC_CORES, e.g. 25,26) are subtracted from the mask so a floater can never preempt a wave worker. This used to be a hand-applied VEX_TASKSET_CORES=1-3,5-24,27 that was easy to forget; it is now done structurally and is idempotent.
Override the whole mask with VEX_TASKSET_CORES=<list> for diagnostics (e.g. to test whether the core layout, not the zero-copy path, drives a behavior).

Why hot tiles are kept off build/relief cores: the deploy discipline is STOP → BUILD → DEPLOY (the node is brought down before a release build — see Build / Deploy), but even so the relief/build cores are physically fenced to CCX0 (1–3) so any incidental background work lands on a separate cache complex and can never thrash the hot pipeline (verify 8–15, replay 16, produce 20, gossip 24) or the wave workers.

The legacy dynamic re-pinner (OFF by default)¶

tools/vex-fd-pin.sh is an older post-launch poller that re-pinned the hottest threads onto a relief pool. It is disabled by default (VEX_DYNAMIC_PIN=0) and is now considered redundant and harmful: with the complete static map in the binary, the poller would migrate the hot replay/gossip tiles off their dedicated cores, fighting the static pins. It exists only as a stopgap (VEX_DYNAMIC_PIN=1) for hardware without a static map. Do not enable it on this layout.

Native QUIC and ports¶

Votes are sent over native QUIC (the QUIC vote poller self-pins to core 28). The explicit ports deploy.sh passes at launch are:

Port / flag	Value	Purpose
`--gossip-port`	`8001`	Gossip / CRDS
`--tvu-port`	`8003`	Turbine shred ingest (the AF_XDP / XSK port)
`--repair-port`	`8002`	Repair
`--rpc-port`	`8899`	JSON-RPC
`--dynamic-port-range`	`8100-8200`	Ephemeral ports (TPU/QUIC ingest, serve) are allocated from this range
`--public-ip`	`38.58.183.154`	Advertised address
`--repair-interface`	the active NIC port	Repair socket bind interface

TPU/QUIC ingest ports are not fixed numbers — they come out of --dynamic-port-range. The cluster filter is set with --expected-shred-version (default 1516) and --expected-genesis-hash (4uhcVJyU9pJkvQyS88uRDiswHXSCkY3zQawwpjk2NsNY); both are overridable at deploy time without a code change (VEX_SHRED_VERSION=<n>).

Confirming the datapath¶

deploy.sh runs two background verifiers after launch. Both are advisory — they report, they never kill the validator.

Tiling verifier — `/tmp/vex-tiling-verify.log`¶

Step 7b-2 reads the live /proc/<pid>/task/*/status affinity masks (actual runtime placement, not code intent) once the replay tile has pinned to core 16. It prints a thread→core histogram and flags any thread that is both unpinned (carrying the wide taskset mask) and busy on a consensus core (recv 5, quic 6, verify 8–15, replay 16, produce 20, gossip 24, plus the wave cores when parallel-exec is armed). A clean run reports no busy floater on a consensus core:

cat /tmp/vex-tiling-verify.log
# === TILING VERIFY (pid=…, replay-tile ready=1) … ===
# thread→core mask histogram (count  mask):
#    …
#   ✅ no busy floater on a consensus core (cores 0-4 walled off via cpuset; recv→5, watchdog→31, floaters→6-24,27)

AF_XDP zero-copy gate¶

Step 7c waits up to 360s for the XSK to bind, then samples the per-queue zero-copy packet counter twice to prove real traffic is flowing on q0. The authoritative bind signal is ss --xdp (the INFO-level "Bound … zero-copy" banner is WARN-filtered out of the live log, so it is a false negative — do not trust its absence):

# Is the XSK actually bound to queue 0?
sudo ss --xdp | grep enp129s0f1np1
# enp129s0f1np1:q0   …

# Is zero-copy traffic actually flowing? (counter should climb)
sudo ethtool -S enp129s0f1np1 | grep -E 'rx0_xsk_packets|rx_xsk_buff_alloc_err'
# rx0_xsk_packets: <climbing>
# rx_xsk_buff_alloc_err: 0      <-- must stay flat

If rx0_xsk_packets climbs and rx_xsk_buff_alloc_err stays flat, zero-copy is engaged correctly.