Environments#
This section describes the system configurations used for experimenting with AiSIO system software architectures, including the initial AiSIO proof-of-concept and the development of a reference implementation. These environments support evaluation, document constraints encountered during development, and serve as reproducible reference builds for readers wishing to explore similar architectures.
The same environments were also used to reproduce, where possible, benchmarks from related work, specifically GDS [2] and BaM [4], ensuring controlled comparison under identical system conditions.
The configurations reflect practical system builds intended to support architectural exploration, implementation, and comparative evaluation, rather than prescribing production deployment guidelines.
Unless otherwise stated, all environments are configured with the platform prerequisites for P2P DMA: Resizable BAR enabled, Above 4G Decoding enabled, IOMMU disabled, and ACS disabled. Note that the availability and naming of these options varies across BIOS vendors; for example, Resizable BAR may appear as “ReBAR”, “Resizable BAR Support”, or “Re-Size BAR Support”.
High-Performance Compute Server#
An enterprise-grade dual-socket server with high-capacity NVMe storage, used for the primary CPU-initiated I/O and benchmark tool comparison experiments. The platform, GPU, and NVMe storage all operate at PCIe Gen5, providing full-bandwidth connectivity throughout the system. With 32 × 64 GiB of DDR5, the system provides 2 TiB of host memory.
Hardware |
Details |
|---|---|
Motherboard |
Dell PowerEdge R760 |
CPU |
2x Intel® Xeon® Gold 6442Y |
Memory |
32x 64GiB Samsung DDR5 4800MHz |
GPU |
1x NVIDIA H100 PCIe 80GB |
Storage |
16x Samsung SSD PM1753 32TB |
NVMe Storage Server#
A dual-socket server with 32 NVMe devices, 16 on each NUMA node, and a GPU on the first node, used for the IOMMU translation overhead experiment. The experiment uses the 16 devices that share a NUMA node with the GPU, so peer-to-peer transfers stay on one socket.
Hardware |
Details |
|---|---|
Motherboard |
Inspur NF5180M6 |
CPU |
2x Intel® Xeon® Scalable 3rd Gen (Ice Lake-SP), 24 cores, SMT |
Memory |
16x 16GiB SK hynix DDR4 2666MHz |
GPU |
1x NVIDIA L4 24GB, PCIe Gen4 x16 |
Storage |
32x Samsung MZTLD15THEPB, PCIe Gen4 x4, 16 per NUMA node |
OS |
Ubuntu 26.04.1 LTS |
The two sockets give 48 cores and 96 threads, with 72MiB of L3 cache in total, and every memory channel holds one module. Each group of eight NVMe devices sits behind a Microchip PCIe Gen4 switch, and each switch connects to its own CPU root port through a Gen4 x16 link, two switches per socket. The GPU has its own Gen4 x16 root port on the first socket. The switches and the GPU limit every link to Gen4. Unlike the other environments, this server boots with the IOMMU off or on depending on the half of the experiment that runs, as described in Experimental Framework.
Professional GPU Workstation#
A workstation built around a professional-grade GPU with a high core-count CPU, used for development and evaluation of the reference implementation. The platform, GPU, and NVMe storage all operate at PCIe Gen4.
Hardware |
Details |
|---|---|
Motherboard |
Supermicro H12SSL-I |
CPU |
1x AMD EPYC 7532 32-Core (Rome) |
Memory |
8x 32GiB Samsung DDR4 2667MHz |
GPU |
1x NVIDIA RTX A5000 24GB |
Storage |
4x Samsung 990 PRO 2TB |
The system supports P2P between all seven PCIe slots, which can be populated with varying combinations of GPUs, NVMe devices, and RDMA NICs, making it an ideal testbed for accelerator-integrated storage I/O topologies.
Desktop Workstation#
A consumer desktop used for development and light experimentation. Both the NVIDIA RTX A2000 6GB and the NVIDIA RTX PRO 2000 Blackwell 16GB have been used and validated in this environment. The AMD B550 platform limits PCIe bandwidth to Gen4 on CPU-direct slots, constraining the RTX PRO 2000 Blackwell below its Gen5 capability. A single NVMe device limits aggregate storage bandwidth and precludes multi-device parallelism.
Hardware |
Details |
|---|---|
Motherboard |
MSI MAG B550M MORTAR WIFI |
CPU |
1x AMD Ryzen 7 5800X 8-Core |
Memory |
2x 16GiB DDR4 2133MHz |
GPU |
1x NVIDIA RTX PRO 2000 Blackwell 16GB / RTX A2000 6GB |
Storage |
1x Samsung 980 PRO 1TB |
Unlike the Professional GPU Workstation, P2P is only possible between devices installed in the PCIe expansion slot labeled PCI_E1 and the M.2 slot labeled M2_1. These are connected to the CPU root complex whereas the other PCIe and M.2 slots are connected to the platform controller hub (PCH).
Development PC#
A desktop PC used to trace how the kernel maps a GPU dma-buf into an NVMe device’s IOMMU domain. Its Intel platform exposes VT-d, so the Intel IOMMU driver’s mapping functions can be probed, and it supports both 2 MiB and 1 GiB pages.
Hardware |
Details |
|---|---|
CPU |
1x Intel® Core™ i9-14900K |
GPU |
1x NVIDIA RTX PRO 4000 Blackwell, 32 GiB BAR1 |
Storage |
1x Samsung 980 PRO, spare disk used for the mapping test |
IOMMU |
Intel VT-d, two units, 2 MiB and 1 GiB pages |
Kernel |
6.8.12-dmabuf, |
The measurement used a cijoe configuration without ssh and a copy of the task retargeted to the spare NVMe, so mappings created and removed during the test did not touch the system disk. Unlike the other environments, this PC boots with the IOMMU off or on depending on the measurement step.
Legacy GPU Server#
A server with legacy Volta-generation GPUs, used for pre-work [19] and the initial AiSIO proof-of-concept [23, 24]. Although the system is PCIe Gen4, the V100 is a PCIe Gen3 device, limiting peak GPU bandwidth to approximately 16 GB/s per slot and constraining peer-to-peer DMA throughput compared to Gen4 systems.
Hardware |
Details |
|---|---|
Motherboard |
Gigabyte G292-Z20 |
CPU |
1x AMD EPYC 7402P 24-Core |
Memory |
8x 32GiB SK Hynix DDR4 2400MHz |
GPU |
2x NVIDIA V100 16GB |
Storage |
4x Samsung 980 PRO 1TB |
The system has four risers, each connected to the motherboard via a PCIe Gen4 x16 slot and carrying a Microsemi PCIe Gen4 switch with two x16 downstream ports. Notably, data can move P2P via the switch without traversing the CPU root complex.