xnvmeperf

xnvmeperf is a multi-threaded async I/O benchmark for NVMe devices. It runs a time-bounded async I/O loop and reports per-device and aggregate IOPS and throughput. A verify subcommand checks data integrity by writing a known per-LBA pattern and reading it back.

When built with CUDA support (XNVME_BE_UPCIE_CUDA_ENABLED), two additional subcommands are available: cuda-run and cuda-verify. These run NVMe I/O directly from CUDA kernels via the uPCIe CUDA backend, bypassing the host I/O path entirely.

Usage: xnvmeperf <command> [<args>]

Where <command> is one of:

  run              | Run a benchmark against the given devices
  verify           | Verify data integrity by writing and reading back a known pattern
  cuda-run         | Run a GPU benchmark against the given devices (requires upcie-cuda backend)
  cuda-verify      | Verify GPU NVMe I/O data integrity (requires upcie-cuda backend)
  htod-roofline    | Host-to-device copy bandwidth ceiling (requires a GPU backend build)

See 'xnvmeperf <command> --help' for the description of [<args>]
See 'xnvmeperf --version' for the library version and the revision it was built from

xnvmeperf - NVMe async IO benchmark -- ver: {major: 0, minor: 7, patch: 5, vcs: '94a52e5'}

run — Benchmark

Runs async I/O against one or more NVMe devices for a fixed duration. Devices are listed as trailing positional arguments. One thread is spawned per CPU in --cpumask, e.g. 0x3 spawns two threads on CPUs 0 and 1. If there are more devices than CPUs, devices are split evenly so each thread owns a slice and runs one async job per owned device. If there are more CPUs than devices, devices are assigned round-robin so multiple threads drive the same device concurrently.

Usage: xnvmeperf run [<uri>...] [<args>]

Run a time-bounded async IO benchmark against one or more NVMe devices.
Devices are distributed across CPU threads pinned by --cpumask or --cpulist.
With --be 'io_uring_dmabuf' the IO buffers are allocated on the GPU given by
--gpu-id and exported as a dma-buf for the kernel to DMA into.
  
Positional arguments:

  [uri ...]                     ; Device URI e.g. '/dev/nvme0n1', '0000:01:00.1', '10.9.8.1:8888', '\\.\PhysicalDrive1'
  
Where <args> include:

  --iopattern STRING            ; IO pattern (read, write, randread, randwrite)
  [ --nqueues NUM ]             ; Number of queues per device
  --qdepth NUM                  ; Use given 'NUM' as queue max capacity
  --iosize NUM                  ; Use given 'NUM' as bs/iosize
  --runtime NUM                 ; Run for 'NUM' seconds
  [ --cpumask STRING ]          ; Hex CPU bitmask for thread pinning (e.g. 0x3)
  [ --cpulist STRING ]          ; Comma-separated list of CPU ids or CPU id ranges for thread pinning (e.g. 0-3,4,6)
  
With <args> for backend:

  [ --be STRING ]               ; xNVMe backend, e.g. 'spdk', 'libvfn', 'upcie'
  [ --subnqn STRING ]           ; Subsystem NQN of the NVMe over Fabrics endpoint e.g. 'nqn.2022-06.io.xnvme:ctrlnode1'
  [ --direct ]                  ; Bypass layers
  [ --poll_io NUM ]             ; For async={io_uring,libaio,kqueue}, enable hipri/io-compl.polling
  [ --poll_sq NUM ]             ; For async=io_uring, enable kernel-side sqthread-poll
  [ --gpu-id NUM ]              ; For be={upcie-cuda,upcie-hip}, GPU ordinal to use
  [ --homi-id NUM ]             ; The HOMI identifier to share controllers under
  [ --report-freq FLOAT ]       ; Report progress every 'FLOAT' seconds
  [ --p2p-cq-mirror ]           ; Complete into GPU memory, beside the data (XNVME_QUEUE_P2P_CQ_MIRROR)
  [ --buf-host-bounce ]         ; Read into a host buffer and copy it to the GPU, instead of P2P
  [ --help ]                    ; Show usage / help

See 'xnvmeperf --help' for other commands

xnvmeperf - NVMe async IO benchmark -- ver: {major: 0, minor: 7, patch: 5, vcs: '94a52e5'}

The title of the results block carries the library version and the revision the library was built from, so a measurement can be traced back to the exact commit. The revision is what git describe --tags --dirty=+ --always reported at build time; a trailing + means the tree had uncommitted changes. A source archive made with meson dist, such as the xnvme-0.7.5.tar.gz CI builds from, carries the revision it was cut from. A copy of the sources with neither git metadata nor that stamp shows unknown in its place.

Example — sequential read on a single device:

xnvmeperf run --iopattern read --qdepth 32 --iosize 4096 \
    --runtime 10 --cpumask 0x1 /dev/nvme0n1

Example — random write across two devices on two CPUs:

xnvmeperf run --iopattern randwrite --qdepth 64 --iosize 4096 \
    --runtime 10 --cpumask 0x3 /dev/nvme0n1 /dev/nvme1n1

With --be upcie-cuda or --be upcie-hip the payloads land in GPU memory while the CPU drives the queues. Adding --p2p-cq-mirror places the completion queues there as well, so the controller’s completion writes no longer queue behind its data writes; see Completions in device memory for when that matters. Example:

xnvmeperf run --iopattern randread --qdepth 128 --iosize 512 \
    --runtime 10 --cpulist 0 --be upcie-cuda --p2p-cq-mirror 0000:01:00.0

For cuda-run and cuda-verify, where the GPU issues the I/O, the queue pair lives in GPU memory. Adding --sq-hostmem moves the submission queue into host memory, which the controller fetches from faster than from GPU memory; see GPU-Resident Queue API for the trade-off.

--buf-host-bounce keeps the I/O on a host backend and moves each payload on to the GPU named by --gpu-id once it completes, with the runtime’s asynchronous copy. Every queue slot has a host buffer of its own, page-locked so the copy runs at link speed, and a slot is reused only once its copy has drained, so when the copy is slower than the drive, submission waits for it. At most 128 MiB of such buffers are in flight per device, so that the heap stays within the GPU’s IOMMU aperture; the host heap is sized with headroom for the per-buffer overhead and for the hugepages a served homi shares, and kept under the 4 GiB at which the DMA mapping collides in that aperture, which is what caps the usable --iosize. A buffer the runtime refuses to page-lock is copied at the pageable rate, which the report then shows. Example:

xnvmeperf run --iopattern randread --qdepth 64 --iosize 4096 \
    --runtime 10 --be upcie --buf-host-bounce --gpu-id 0 0000:01:00.0

htod-roofline measures what that copy delivers on its own: it keeps --qdepth page-locked copies of --iosize in flight to the GPU for --runtime seconds and reports the host-to-device GB/s, taking no devices:

xnvmeperf htod-roofline --iosize 4096 --qdepth 64 --runtime 10 --gpu-id 0

verify — Data integrity check

Writes --count sequential I/Os with a per-LBA stamp pattern, then reads them back and compares against the expected data. Reports the number of mismatches and I/O errors per device, and exits non-zero when either is not zero. On a namespace whose metadata is kept apart from the data, the metadata is written and compared too, unless the backend takes no metadata buffer.

Usage: xnvmeperf verify [<uri>...] [<args>]

For each device: writes --count sequential IOs with a per-LBA pattern,
then reads them back and compares against the expected data.
  
Positional arguments:

  [uri ...]                     ; Device URI e.g. '/dev/nvme0n1', '0000:01:00.1', '10.9.8.1:8888', '\\.\PhysicalDrive1'
  
Where <args> include:

  --iosize NUM                  ; Use given 'NUM' as bs/iosize
  --count NUM                   ; Use given 'NUM' as count
  
With <args> for backend:

  [ --be STRING ]               ; xNVMe backend, e.g. 'spdk', 'libvfn', 'upcie'
  [ --subnqn STRING ]           ; Subsystem NQN of the NVMe over Fabrics endpoint e.g. 'nqn.2022-06.io.xnvme:ctrlnode1'
  [ --direct ]                  ; Bypass layers
  [ --poll_io NUM ]             ; For async={io_uring,libaio,kqueue}, enable hipri/io-compl.polling
  [ --poll_sq NUM ]             ; For async=io_uring, enable kernel-side sqthread-poll
  [ --gpu-id NUM ]              ; For be={upcie-cuda,upcie-hip}, GPU ordinal to use
  [ --homi-id NUM ]             ; The HOMI identifier to share controllers under
  [ --p2p-cq-mirror ]           ; Complete into GPU memory, beside the data (XNVME_QUEUE_P2P_CQ_MIRROR)
  [ --help ]                    ; Show usage / help

See 'xnvmeperf --help' for other commands

xnvmeperf - NVMe async IO benchmark -- ver: {major: 0, minor: 7, patch: 5, vcs: '94a52e5'}

Example:

xnvmeperf verify --iosize 4096 --count 256 /dev/nvme0n1

cuda-run — GPU benchmark

See also

uPCIe CUDA

Backend setup, system configuration, and memory architecture.

GPU IOMMU domain

Required with an IOMMU enabled. Without it the run succeeds while every I/O fails.

GPU-Resident Queue API

The libxnvme_cuda API used to create queues and dispatch commands from CUDA kernels.

Requires the upcie-cuda backend. All queues across all devices are driven by a single CUDA kernel: each CUDA block owns one NVMe queue and each thread within the block owns one queue slot. The grid has ndevs × --nqueues blocks in total.

The slots of a queue form groups of one warp, 32 slots, that take turns: a group reaps the queue’s next batch of completions and refills the room they leave, while the other groups’ commands stay in service. Submitting the whole depth and then waiting for it would leave the queue empty for one GPU round trip per depth’s worth of I/O, so a single queue per drive could not reach the drive’s rate however deep it is. A depth below 32 is a single group.

A slot’s previous command may still be in flight when a group reuses it, since a group reaps the queue’s oldest completions and not necessarily its own, so a command identifier advances by the depth on every submission. Each thread fences its own entry before the group’s barrier: with two batches in flight, a single fence by the thread that rings the doorbell let the controller fetch entries before they were written and execute what it found there, a few per million. The stop flag lives in host memory, where reading it is a PCIe round trip on the turn’s critical path, so the kernel samples it every 32nd turn, and it rings the doorbell without a second fence, since every entry was fenced before the barrier.

Both --qdepth and --iosize must be powers of 2. Supported patterns are read, write, randread, and randwrite.

Usage: xnvmeperf cuda-run [<uri>...] [<args>]

Run a time-bounded GPU NVMe I/O benchmark against one or more devices.
All devices run in parallel: one CUDA block per queue, all queues launched
in a single kernel. All devices must use the same LBA size.
  
Positional arguments:

  [uri ...]                     ; Device URI e.g. '/dev/nvme0n1', '0000:01:00.1', '10.9.8.1:8888', '\\.\PhysicalDrive1'
  
Where <args> include:

  --iopattern STRING            ; IO pattern (read, write, randread, randwrite)
  [ --nqueues NUM ]             ; Number of queues per device
  --qdepth NUM                  ; Use given 'NUM' as queue max capacity
  --iosize NUM                  ; Use given 'NUM' as bs/iosize
  --runtime NUM                 ; Run for 'NUM' seconds
  
With <args> for backend:

  [ --be STRING ]               ; xNVMe backend, e.g. 'spdk', 'libvfn', 'upcie'
  [ --gpu-id NUM ]              ; For be={upcie-cuda,upcie-hip}, GPU ordinal to use
  [ --homi-id NUM ]             ; The HOMI identifier to share controllers under
  [ --report-freq FLOAT ]       ; Report progress every 'FLOAT' seconds
  [ --sq-hostmem ]              ; Submit GPU-issued I/O from host memory (XNVME_QUEUE_SQ_HOSTMEM)
  [ --help ]                    ; Show usage / help

See 'xnvmeperf --help' for other commands

xnvmeperf - NVMe async IO benchmark -- ver: {major: 0, minor: 7, patch: 5, vcs: '94a52e5'}

Example — sequential read, four queues of depth 32 on two devices:

xnvmeperf cuda-run --iopattern read --nqueues 4 --qdepth 32 --iosize 4096 \
    --runtime 10 --be upcie-cuda 0000:01:00.0 0000:02:00.0

Example — random write, single queue:

xnvmeperf cuda-run --iopattern randwrite --qdepth 64 --iosize 4096 \
    --runtime 10 --be upcie-cuda 0000:01:00.0

cuda-verify — GPU data integrity check

Uses the same queue topology as cuda-run. For each queue slot the kernel writes a unique LBA-stamped pattern, then reads it back on the host and compares against the expected data. This is intended to confirm that cuda-run results reflect correct I/O rather than silent data corruption.

Usage: xnvmeperf cuda-verify [<uri>...] [<args>]

Write an LBA-stamped pattern to each device through GPU queues and read
it back, verifying that the data matches. Uses the same queue topology
as cuda-run so results are directly comparable.
  
Positional arguments:

  [uri ...]                     ; Device URI e.g. '/dev/nvme0n1', '0000:01:00.1', '10.9.8.1:8888', '\\.\PhysicalDrive1'
  
Where <args> include:

  --iosize NUM                  ; Use given 'NUM' as bs/iosize
  [ --nqueues NUM ]             ; Number of queues per device
  --qdepth NUM                  ; Use given 'NUM' as queue max capacity
  
With <args> for backend:

  [ --be STRING ]               ; xNVMe backend, e.g. 'spdk', 'libvfn', 'upcie'
  [ --gpu-id NUM ]              ; For be={upcie-cuda,upcie-hip}, GPU ordinal to use
  [ --homi-id NUM ]             ; The HOMI identifier to share controllers under
  [ --sq-hostmem ]              ; Submit GPU-issued I/O from host memory (XNVME_QUEUE_SQ_HOSTMEM)
  [ --help ]                    ; Show usage / help

See 'xnvmeperf --help' for other commands

xnvmeperf - NVMe async IO benchmark -- ver: {major: 0, minor: 7, patch: 5, vcs: '94a52e5'}

Example:

xnvmeperf cuda-verify --iosize 4096 --qdepth 32 --be upcie-cuda 0000:01:00.0