Experimental Framework

This section describes the system configuration, software stack, and benchmark workflows used to produce the experimental results. It covers both the synthetic NVMe benchmarks that characterize raw I/O performance and the file-based benchmarks that evaluate end-to-end dataset loading.

System Setup

All environment provisioning is automated using CIJOE, a tool for systems development and testing. CIJOE runs on a local initiator machine and connects to a remote target over SSH. Shell commands and Python scripts are collected into YAML workflow definitions that document the execution sequence and combined purpose. Input values are kept separate from scripts in TOML configuration files, so the same workflow can be replicated across different environments without modifying the scripts themselves. After execution, CIJOE generates a report covering command output, script documentation, collected artifacts, and a workflow summary.

Step-by-step instructions for running the CIJOE setup tasks are provided in the README.md of the AiSIO repository. The table below lists all software components installed on each system, in installation order. It covers both the proof-of-concept stack (proprietary NVIDIA GPUDirect-Storage / BaM, reproducible at the poc tag) and the current 26.04 open stack (NVIDIA and AMD over the udmabuf-import path); rows specific to the latter are tagged “26.04 stack” in the description.

Category	Component	Version	Install	Description
OS	Ubuntu Server	24.04.4	ISO	Base operating system; GA kernel required (HWE must be disabled)
Kernel	Linux + udmabuf-import	6.8 GA + patch	Source	Ubuntu GA kernel patched to add DMA-buf importer support to UDMABUF, enabling physical address resolution of device memory from user space
NVIDIA	MLNX_OFED	24.10-3.2.5.0	Source	Mellanox OFED with NVMe-oF, NFS-RDMA, and GPUDirect Storage support
NVIDIA	Open Kernel Modules	570.x	APT	NVIDIA open-source GPU kernel driver
NVIDIA	CUDA Toolkit	12.8.x	APT	CUDA compiler, runtime, and libraries
NVIDIA	nvidia-fs	12.8.x	APT	GPUDirect Storage kernel module
OS	Ubuntu Server	26.04	ISO	26.04 stack: current base OS (GA 7.0 kernel)
Kernel	Linux + udmabuf-import	7.0 GA + patch	Source	26.04 stack: GA 7.0 kernel patched with the UDMABUF importer
NVIDIA	nvidia-driver-580-open	580 (open)	APT	26.04 stack: open-kernel-module driver from the distro ‘restricted’ repo
NVIDIA	CUDA Toolkit	13	APT	26.04 stack: CUDA from NVIDIA’s ubuntu2604 repo
AMD	ROCm (HIP + HSA)	7.1	APT	26.04 stack: ROCm userspace from 26.04 ‘universe’; in-tree amdgpu, no DKMS
AiSIO	fio	3.41	Source	I/O benchmark tool; built from source for SPDK plugin integration
AiSIO	SPDK	v26.01	Source	Storage Performance Development Kit; provides the user space NVMe driver and bdevperf
AiSIO	xNVMe	main	Source	Unified NVMe command interface with uPCIe backend support
AiSIO	xal	v0.3.1	Source	XFS Abstraction Library for file-to-block extent resolution
AiSIO	fil	v0.3.1	Source	File Iterator Library; benchmark harness used to evaluate different I/O backends
Tools	devbind	—	pipx	NVMe PCIe driver binding utility
Tools	hugepages	—	pipx	Huge page allocation management utility

Benchmarks

The benchmarks fall into two groups: synthetic benchmarks that issue I/O directly against raw NVMe devices, and file-based benchmarks that load datasets from an XFS filesystem on a dedicated NVMe device.

Synthetic

Synthetic benchmarks issue I/O directly to NVMe devices via user space drivers, bypassing the kernel filesystem stack. They require a device configuration file listing the NVMe block devices to use. configs/devices_16.toml is provided as an example and must be edited to match the target system. Device PCI addresses can be found with:

lspci | grep Non-Volatile

If the boot device is an NVMe device, it must be excluded from driver rebinding by setting the xnvme.driver.prefix key in the configuration file:

[xnvme.driver]
prefix = "PCI_BLACKLIST=0000:01:00.0"

Devices must be unbound from the kernel NVMe driver and bound to uio_pci_generic. Huge pages are also required, as both SPDK and xNVMe uPCIe rely on DMA-capable memory that must be physically contiguous and pinned:

devbind --device '<pci_addr>' --bind uio_pci_generic
hugepages setup --count 1024

The benchmark workflows are parameterised by editing the run step in the respective task file. The keys under with correspond to independent variables. numcpus_range and numdevs_range are inclusive tuples defining the range of values tested, not complete lists. An optional results_dir key enables continuation of a previous run, and repetitions controls the number of runs per configuration (default: 5). When re-running benchmarks without changing system state, the hugepage allocation and driver binding steps can be skipped by specifying only the steps to execute:

cijoe \
    -c configs/transport.toml \
    -c configs/aisio.toml \
    -c configs/devices_16.toml \
    tasks/bench_io.yaml \
    run combine visualize

Results are rendered as an interactive HTML page at cijoe-output/artifacts/benchmark-results.html, which allows results from different parameterizations to be compared.

CPU-initiated I/O (bench_io.yaml)

Characterizes the maximum IOPS achievable through CPU-driven I/O using SPDK’s bdevperf across a wide parameter space. Described in detail in CPU-initiated I/O: Optimal Parameter Search.

cijoe --monitor \
    -c configs/transport.toml \
    -c configs/aisio.toml \
    -c configs/devices_16.toml \
    tasks/bench_io.yaml

CPU-initiated I/O: Software Abstraction Overhead (bench_tools.yaml)

Runs bdevperf, SPDK NVMe Perf, and xnvmeperf (with the spdk, upcie, and upcie-cuda backends) under identical parameters to isolate the effect of tool and driver implementation on measured IOPS. Described in detail in CPU-initiated I/O: Software Abstraction Overhead.

cijoe --monitor \
    -c configs/transport.toml \
    -c configs/aisio.toml \
    -c configs/devices_16.toml \
    tasks/bench_tools.yaml

CPU-Initiated P2P I/O: PCIe Bandwidth Saturation (bench_pcie.yaml)

Characterizes PCIe link utilization on the upcie-cuda path by running xnvmeperf at varying I/O sizes while collecting hardware-level PCIe bandwidth counters via DCGM and a reference P2P bandwidth measurement from p2pBandwidthLatencyTest. Described in detail in CPU-Initiated P2P I/O: PCIe Bandwidth Saturation.

The p2pBandwidthLatencyTest reference binary is built by setup_nvstack.yaml at a fixed path, and DCGM fields default in the collector, so no extra config is required.

cijoe --monitor \
    -c configs/transport.toml \
    -c configs/aisio.toml \
    -c configs/devices_16.toml \
    tasks/bench_pcie.yaml

Device-initiated I/O: I/O Size Scaling (bench_cuda_iosize.yaml)

Characterizes the minimum CUDA thread count needed to saturate the PCIe link under device-initiated I/O, using xnvmeperf with the cuda-run subcommand and the upcie-cuda backend, with queue depth as the secondary variable. Described in detail in Device-initiated I/O: I/O Size Scaling.

The p2pBandwidthLatencyTest reference binary is built by setup_nvstack.yaml at a fixed path, so no extra config is required.

cijoe --monitor \
    -c configs/transport.toml \
    -c configs/aisio.toml \
    -c configs/devices_16.toml \
    tasks/bench_cuda_iosize.yaml

Device-initiated I/O: Queue Depth Scaling (bench_cuda_qdepth.yaml)

Characterizes how IOPS scales with queue depth under device-initiated I/O, using xnvmeperf with the cuda-run subcommand and the upcie-cuda backend, with the number of queues per device as the secondary variable. Described in detail in Device-initiated I/O: Queue Depth Scaling.

cijoe --monitor \
    -c configs/transport.toml \
    -c configs/aisio.toml \
    -c configs/devices_16.toml \
    tasks/bench_cuda_qdepth.yaml

File-based

File-based benchmarks load datasets from an XFS filesystem on a dedicated NVMe device. The device and mount point are specified in configs/datasets.toml. The following workflow formats the device, mounts it, and generates the synthetic datasets, and runs an XFS defragmentation pass to ensure contiguous file layout:

cijoe --monitor \
    -c configs/transport.toml \
    -c configs/datasets.toml \
    tasks/setup_dataset.yaml

Three datasets are generated, each modeling a different real-world workload in terms of file count and file size distribution:

imagenetish models an image classification dataset. It consists of 1000 classes with 1000 to 1400 files each (1.0–1.4 million files total), with individual file sizes between 75 KB and 150 KB.

tiktokish models a short-form video library. It consists of 32 classes with 14 files each (448 files total), with individual file sizes between 7 MiB and 8 MiB.

filesize8gib models a large-file workload. It consists of 20 files, each exactly 8 GiB, used to evaluate performance with large individual files.

All datasets are generated with a fixed random seed to ensure reproducibility across systems.

The AiSIO benchmark uses the xNVMe uPCIe path and requires the device bound to uio_pci_generic with hugepages allocated:

umount /mnt/datasets
devbind --device '<pci_addr>' --bind uio_pci_generic
hugepages setup --count 1024

AiSIO / uPCIe (bench_aisio.yaml)

This benchmark consists of two parts. First, all three datasets (tiktokish, imagenetish, filesize8gib) are loaded through FIL over the AiSIO path using the aisio-cpu, aisio-gpu, and aisio-p2p backends, measuring end-to-end dataset-loading performance for CPU-driven, GPU-driven, and peer-to-peer transfers. Second, synthetic random-read benchmarks are run with xnvmeperf using the upcie (CPU completion) and upcie-cuda (GPU completion) backends, measuring raw I/O throughput on the uPCIe path.

cijoe --monitor \
    -c configs/transport.toml \
    -c configs/aisio.toml \
    -c configs/datasets.toml \
    tasks/bench_aisio.yaml

POSIX (bench_posix.yaml)

This proof-of-concept benchmark provided a POSIX I/O baseline by loading all three datasets (imagenetish, tiktokish, filesize8gib) through the posix backend of FIL, for comparison against the AiSIO path. It is reproducible only from the poc tag, where bench_posix.yaml lives; the current open stack does not include it.

NVIDIA GPUDirect Storage (bench_gds.yaml)

This proof-of-concept benchmark evaluated NVIDIA GPUDirect Storage through the FIL interface, loading all three datasets (imagenetish, tiktokish, filesize8gib) through the gds backend and running synthetic sequential and random-read benchmarks with gdsio. It belongs to the proprietary GDS path and is reproducible only from the poc tag, where bench_gds.yaml and its gdsio wrapper live; the current open stack does not include it.

GDS Transfer Mode Comparison (compare_gds_xfermodes.yaml)

This proof-of-concept workflow used gdsio to compare GDS transfer types (XferType) for small 4 KiB I/O, identifying which transfer mode best suited small-I/O workloads. Like the GDS benchmark above, it is reproducible only from the poc tag.