Experimental Framework#

This section describes the system configuration, software stack, and benchmark workflows used to produce the experimental results. It covers both the synthetic NVMe benchmarks that characterize raw I/O performance and the file-based benchmarks that evaluate end-to-end dataset loading.

System Setup#

All environment provisioning is automated using CIJOE, a tool for systems development and testing. CIJOE runs on a local initiator machine and connects to a remote target over SSH. Shell commands and Python scripts are collected into YAML workflow definitions that document the execution sequence and combined purpose. Input values are kept separate from scripts in TOML configuration files, so the same workflow can be replicated across different environments without modifying the scripts themselves. After execution, CIJOE generates a report covering command output, script documentation, collected artifacts, and a workflow summary.

Step-by-step instructions for running the CIJOE setup tasks are provided in the README.md of the AiSIO repository. The table below lists all software components installed on each system, in installation order. It covers both the proof-of-concept stack (proprietary NVIDIA GPUDirect-Storage / BaM, reproducible at the poc tag) and the current 26.04 open stack (NVIDIA and AMD over the dma-buf import path); rows specific to the latter are tagged “26.04 stack” in the description.

Category	Component	Version	Install	Description
OS	Ubuntu Server	24.04.4	ISO	Base operating system; GA kernel required (HWE must be disabled)
Kernel	Linux + udmabuf-import	6.8 GA + patch	Source	Ubuntu GA kernel patched to add DMA-buf importer support to UDMABUF, enabling physical address resolution of device memory from user space
NVIDIA	MLNX_OFED	24.10-3.2.5.0	Source	Mellanox OFED with NVMe-oF, NFS-RDMA, and GPUDirect Storage support
NVIDIA	Open Kernel Modules	570.x	APT	NVIDIA open-source GPU kernel driver
NVIDIA	CUDA Toolkit	12.8.x	APT	CUDA compiler, runtime, and libraries
NVIDIA	nvidia-fs	12.8.x	APT	GPUDirect Storage kernel module
OS	Ubuntu Server	26.04	ISO	26.04 stack: current base OS (GA 7.0 kernel)
Kernel	upcie-experimental	0.9.0	DKMS	26.04 stack: out-of-tree modules released by uPCIe; the dma-buf importer and the IOMMU physical-address mapper
NVIDIA	nvidia-driver-580-open	580 (open)	APT	26.04 stack: open-kernel-module driver from the distro ‘restricted’ repo
NVIDIA	CUDA Toolkit	13	APT	26.04 stack: CUDA from NVIDIA’s ubuntu2604 repo
NVIDIA	DCGM	4.x (cuda13)	APT	26.04 stack: GPU telemetry (nv-hostengine + dcgmi); collects the GPU metrics reported by the benchmark workflows
AMD	ROCm (HIP + HSA)	7.1	APT	26.04 stack: ROCm userspace from 26.04 ‘universe’; in-tree amdgpu, no DKMS
AiSIO	fio	3.41	Source	I/O benchmark tool; built from source for SPDK plugin integration
AiSIO	SPDK	v26.01	Source	Storage Performance Development Kit; provides the user space NVMe driver and bdevperf
AiSIO	xNVMe	main	Source	Unified NVMe command interface with uPCIe backend support
AiSIO	xal	v0.3.2	Source	XFS Abstraction Library for file-to-block extent resolution
AiSIO	fil	v0.3.3	Source	File Iterator Library; benchmark harness used to evaluate different I/O backends
Tools	devbind	—	pipx	NVMe PCIe driver binding utility
Tools	hugepages	—	pipx	Huge page allocation management utility

Benchmarks#

The benchmarks fall into two groups: synthetic benchmarks that issue I/O directly against raw NVMe devices, and file-based benchmarks that load datasets from an XFS filesystem on a dedicated NVMe device.

Synthetic#

Synthetic benchmarks issue I/O directly to NVMe devices via user space drivers, bypassing the kernel filesystem stack. They require a device configuration file listing the NVMe block devices to use. configs/devices_16.toml is provided as an example and must be edited to match the target system. Device PCI addresses can be found with:

lspci | grep Non-Volatile

If the boot device is an NVMe device, it must be excluded from driver rebinding by setting the xnvme.driver.prefix key in the configuration file:

[xnvme.driver]
prefix = "PCI_BLACKLIST=0000:01:00.0"

Devices must be unbound from the kernel NVMe driver and bound to uio_pci_generic. Huge pages are also required, as both SPDK and xNVMe uPCIe rely on DMA-capable memory that must be physically contiguous and pinned:

devbind --device '<pci_addr>' --bind uio_pci_generic
hugepages setup --count 1024

Device Fill State#

Every namespace is formatted before synthetic benchmarks are run, which leaves all logical blocks deallocated. Reads of deallocated blocks are served by the controller without media access resulting in a synthetically low latency. The figures this produces may therefore exceed those seen in real-world, data-bearing workloads. They are an upper bound on what the I/O path can drive, and are to be read as a measure of system overhead and scalability rather than as an indicator of application-level storage performance.

The format is a manual step rather than a workflow step, because it only has to be done once: every synthetic benchmark issues randread exclusively, so none of them write to the devices and the state established by the format survives across runs. Formatting goes through the kernel NVMe driver, so it precedes the binding to uio_pci_generic above and each device is rebound afterwards:

devbind --device '<pci_addr>' --bind nvme
nvme format /dev/<ns> --force
devbind --device '<pci_addr>' --bind uio_pci_generic

GPU Telemetry#

The benchmarks that transfer into GPU memory collect GPU telemetry with dcgmi dmon alongside what the benchmark tool itself reports. The fields are those of the DCGM feature overview, which defines each of them; every experiment lists the ones it reports.

Fields are sampled every 100 ms and reported as mean/p95/min/max over the samples in which the devices were transferring. A sample counts as transferring when the PCIe receive traffic (field 1010) exceeds 100 MB/s or the graphics engine (field 1001) is more than 1% active. The monitoring window also spans process startup and teardown, whose duration varies with the configuration under test, so a mean over the whole window would describe the length of the setup as much as the workload. The share of the window that qualified is recorded alongside the statistics.

The engine fields report what is running on the GPU: SM activity (1002) how much of the device has a warp resident on it, warp slot occupancy (1003) how much of the warp capacity of a multiprocessor is taken, and DRAM_ACTIVE (1005) how much of the GPU memory bandwidth is in use.

The clocks (fields 100 and 101), the throttle reason bits (112), the PCIe replay counter (202) and the link generation and width (237/238) are collected as guards on whether runs are comparable: the activity fields are fractions of cycles, so a run measured at a sagging clock cannot be held against one measured at boost, and a downtrained or retransmitting link changes the bandwidth there is to reach.

Running the Workflows#

The benchmark workflows are parameterised by editing the run step in the respective task file. The keys under with correspond to independent variables. numcpus_range and numdevs_range are inclusive tuples defining the range of values tested, not complete lists. An optional results_dir key enables continuation of a previous run, and repetitions controls the number of runs per configuration (default: 5). When re-running benchmarks without changing system state, the driver binding step can be skipped by specifying only the steps to execute:

cijoe \
    -c configs/transport.toml \
    -c configs/aisio.toml \
    -c configs/devices_16.toml \
    tasks/bench_io.yaml \
    run combine visualize

Results are rendered as an interactive HTML page at cijoe-output/artifacts/benchmark-results.html, which allows results from different parameterizations to be compared.

CPU-initiated I/O (`bench_io.yaml`)#

Characterizes the maximum IOPS achievable through CPU-driven I/O using SPDK’s bdevperf across a wide parameter space. Described in detail in CPU-initiated I/O: Optimal Parameter Search.

cijoe --monitor \
    -c configs/transport.toml \
    -c configs/aisio.toml \
    -c configs/devices_16.toml \
    tasks/bench_io.yaml

CPU-initiated I/O: Software Abstraction Overhead (`bench_tools.yaml`)#

Runs bdevperf, SPDK NVMe Perf, and xnvmeperf (with the spdk, upcie, and upcie-cuda backends) under identical parameters to isolate the effect of tool and driver implementation on measured IOPS. Described in detail in CPU-initiated I/O: Software Abstraction Overhead.

cijoe --monitor \
    -c configs/transport.toml \
    -c configs/aisio.toml \
    -c configs/devices_16.toml \
    tasks/bench_tools.yaml

CPU-Initiated P2P I/O: PCIe Bandwidth Saturation (`bench_pcie.yaml`)#

Characterizes PCIe link utilization on the upcie-cuda path by running xnvmeperf at varying I/O sizes while collecting hardware-level PCIe bandwidth counters via DCGM and a reference PCIe bandwidth measurement from nvbandwidth. Described in detail in CPU-Initiated P2P I/O: PCIe Bandwidth Saturation.

nvbandwidth copies from host memory into GPU memory across the PCIe link of the GPU given by dcgm.gpu, the same link the NVMe devices transfer over under P2P, which measures the peak bandwidth that link sustains. The value recorded is its host_to_device_memcpy_ce result.

This and bench_cuda_iosize.yaml, measure the PCIe bandwidth with nvbandwidth as a step of their own run rather than carrying a fixed value across experiments. As such, the experiments can quote figures that differ slightly in the last digit.

cijoe --monitor \
    -c configs/transport.toml \
    -c configs/aisio.toml \
    -c configs/devices_16.toml \
    tasks/bench_pcie.yaml

Device-initiated I/O: I/O Size Scaling (`bench_cuda_iosize.yaml`)#

Characterizes the minimum CUDA thread count needed to saturate the PCIe link under device-initiated I/O, using xnvmeperf with the cuda-run subcommand and the upcie-cuda backend, with queue depth as the secondary variable. Described in detail in Device-initiated I/O: I/O Size Scaling.

cijoe --monitor \
    -c configs/transport.toml \
    -c configs/aisio.toml \
    -c configs/devices_16.toml \
    tasks/bench_cuda_iosize.yaml

Device-initiated I/O: Queue Depth Scaling (`bench_cuda_qdepth.yaml`)#

Characterizes how IOPS scales with queue depth under device-initiated I/O, using xnvmeperf with the cuda-run subcommand and the upcie-cuda backend, with the number of queues per device as the secondary variable. Described in detail in Device-initiated I/O: Queue Depth Scaling.

cijoe --monitor \
    -c configs/transport.toml \
    -c configs/aisio.toml \
    -c configs/devices_16.toml \
    tasks/bench_cuda_qdepth.yaml

File-based#

File-based benchmarks load datasets from an XFS filesystem on a dedicated NVMe device. The device and mount point are specified in configs/datasets.toml. The following workflow formats the device, mounts it, and generates the synthetic datasets, and runs an XFS defragmentation pass to ensure contiguous file layout:

cijoe --monitor \
    -c configs/transport.toml \
    -c configs/datasets.toml \
    tasks/setup_dataset.yaml

Three datasets are generated, each modeling a different real-world workload in terms of file count and file size distribution:

imagenetish models an image classification dataset. It consists of 1000 classes with 1000 to 1400 files each (1.0–1.4 million files total), with individual file sizes between 75 KB and 150 KB.

tiktokish models a short-form video library. It consists of 32 classes with 14 files each (448 files total), with individual file sizes between 7 MiB and 8 MiB.

filesize8gib models a large-file workload. It consists of 20 files, each exactly 8 GiB, used to evaluate performance with large individual files.

All datasets are generated with a fixed random seed to ensure reproducibility across systems.

The AiSIO benchmark uses the xNVMe uPCIe path and requires the device bound to uio_pci_generic with hugepages allocated:

umount /mnt/datasets
devbind --device '<pci_addr>' --bind uio_pci_generic
hugepages setup --count 1024

AiSIO / uPCIe (`bench_aisio.yaml`)#

This benchmark consists of two parts. First, all three datasets (tiktokish, imagenetish, filesize8gib) are loaded through FIL over the AiSIO path using the aisio-cpu, aisio-gpu, and aisio-p2p backends, measuring end-to-end dataset-loading performance for CPU-driven, GPU-driven, and peer-to-peer transfers. Second, synthetic random-read benchmarks are run with xnvmeperf using the upcie (CPU completion) and upcie-cuda (GPU completion) backends, measuring raw I/O throughput on the uPCIe path.

cijoe --monitor \
    -c configs/transport.toml \
    -c configs/aisio.toml \
    -c configs/datasets.toml \
    tasks/bench_aisio.yaml

POSIX (`bench_posix.yaml`)#

This proof-of-concept benchmark provided a POSIX I/O baseline by loading all three datasets (imagenetish, tiktokish, filesize8gib) through the posix backend of FIL, for comparison against the AiSIO path. It is reproducible only from the poc tag, where bench_posix.yaml lives; the current open stack does not include it.

NVIDIA GPUDirect Storage (`bench_gds.yaml`)#

This proof-of-concept benchmark evaluated NVIDIA GPUDirect Storage through the FIL interface, loading all three datasets (imagenetish, tiktokish, filesize8gib) through the gds backend and running synthetic sequential and random-read benchmarks with gdsio. It belongs to the proprietary GDS path and is reproducible only from the poc tag, where bench_gds.yaml and its gdsio wrapper live; the current open stack does not include it.

GDS Transfer Mode Comparison (`compare_gds_xfermodes.yaml`)#

This proof-of-concept workflow used gdsio to compare GDS transfer types (XferType) for small 4 KiB I/O, identifying which transfer mode best suited small-I/O workloads. Like the GDS benchmark above, it is reproducible only from the poc tag.

Experimental Framework

Contents

Experimental Framework#

System Setup#

Benchmarks#

Synthetic#

Device Fill State#

GPU Telemetry#

Running the Workflows#

CPU-initiated I/O (bench_io.yaml)#

CPU-initiated I/O: Software Abstraction Overhead (bench_tools.yaml)#

CPU-Initiated P2P I/O: PCIe Bandwidth Saturation (bench_pcie.yaml)#

Device-initiated I/O: I/O Size Scaling (bench_cuda_iosize.yaml)#

Device-initiated I/O: Queue Depth Scaling (bench_cuda_qdepth.yaml)#