Skip to content

Running with AF_XDP

Docs-site note: A getting-started guide for turning AF_XDP on. For the deep reference — queue/IRQ tuning, the mlx5 driver knobs, and the full core-pinning layout — see Network Setup, which this page deliberately does not restate in full.

AF_XDP is Vexor's zero-copy networking fast path. It's optional, opt-in, and — like everything on this site — described here exactly as it behaves, not as it might one day behave.

Receive-only, by design

Vexor's AF_XDP path accelerates receive (RX) only. Transmit (TX) — sending shreds, votes, and repair responses — always goes through the standard kernel UDP send path. Don't document or assume AF_XDP zero-copy on the send side; it isn't there.

What it does

On the receive side, AF_XDP lets the NIC DMA incoming shreds and packets straight into a userspace ring buffer (an XSK), skipping the per-packet kernel-stack copy that a normal UDP socket incurs. At the packet rate a validator's shred ingest sees, that copy is real, measurable overhead — AF_XDP removes it for RX.

The default deploy path, with AF_XDP off, is kernel-UDP — the proven-safe baseline. AF_XDP is the recommended high-throughput option when the hardware supports it, and it's what Vexor's own live testnet node runs.

Prerequisites

  • NIC and driver. Vexor's AF_XDP path is proven on a Mellanox ConnectX-6 Dx using the mlx5 driver, which has mature AF_XDP/XDP support and hardware queues that map cleanly onto a dedicated receive core. Other AF_XDP-capable NICs/drivers may work but are not what Vexor's reference deployment runs on — see Hardware for the reference box.
  • Kernel capabilities. AF_XDP needs cap_net_raw, cap_net_admin, and cap_bpf (plus cap_perfmon for the BPF/XDP tooling Vexor's deploy path uses), a raised locked-memory limit (ulimit -l) for the UMEM frame pool, and unprivileged BPF enabled. deploy.sh sets these automatically on Vexor's reference deployment; see Network Setup: Prerequisites for AF_XDP for the exact settings and how to check them by hand.
  • A dedicated receive core and queue. AF_XDP performs best with its own CPU core and its own NIC hardware queue, not shared with the general taskset. The full core-map and queue-steering setup is covered in Network Setup.

Enabling it

The operator-facing switch is one environment variable:

VEX_ENABLE_AFXDP=1 bash deploy.sh dev

Under the hood, deploy.sh translates VEX_ENABLE_AFXDP=1 into the two launch flags that actually select the AF_XDP transport:

--enable-af-xdp --xdp-zero-copy

Those two CLI flags — not the environment variable by itself — are what the binary checks to decide whether to bring up the zero-copy ring. VEX_ENABLE_AFXDP itself additionally gates two secondary behaviors read directly by the binary: a dedicated jemalloc arena for the receive thread, and AF_XDP-specific instrumentation logging. If you're launching vex-fd directly rather than through deploy.sh, pass --enable-af-xdp --xdp-zero-copy yourself.

Leave VEX_ENABLE_AFXDP unset (or =0) to stay on the kernel-UDP path — nothing else changes; AF_XDP is purely additive.

Verifying it worked

  • Boot log. With AF_XDP enabled, the deploy sequence's final health check confirms the zero-copy socket actually bound, in addition to the normal "process alive, threads landed on pinned cores" checks — see Deploying Vexor.
  • Metrics. Once running, the vexor-afxdp measurement set (rx_dropped, rx_invalid_descs, tx_invalid_descs, rx_ring_full, rx_fill_ring_empty_descs, tx_ring_empty_descs) is read straight from the kernel via getsockopt(SOL_XDP, XDP_STATISTICS) and only appears in your metrics stream when AF_XDP is active. Seeing it appear (with low/zero drop and ring-full counters) is a good sign. See Observability.
  • Queue check. Confirm the AF_XDP socket bound to the queue you steered traffic to (queue 0 in Vexor's reference layout) — see Network Setup.

Troubleshooting

  • AF_XDP silently falls back to copy mode, or fails to bind. This is the expected behavior when a prerequisite is missing (capability, locked-memory limit, unprivileged BPF, or driver support) rather than a crash — check each prerequisite in Network Setup: Prerequisites for AF_XDP.
  • No vexor-afxdp metrics appear. Either AF_XDP isn't actually active (double-check the boot log for the zero-copy-socket-bound confirmation) or metrics aren't configured at all — see Observability.
  • General boot/deploy failures. See Troubleshooting for the broader deploy-failure playbook.

See also

  • Network Setup — the full tuning reference: queue/IRQ steering, mlx5 driver knobs, and the core-pinning layout AF_XDP shares with the rest of the validator.
  • Hardware — the reference NIC and box.
  • Observability — the vexor-afxdp measurement set.