CPU-Initiated P2P I/O: PCIe Bandwidth Saturation

The CPU-initiated I/O: Software Abstraction Overhead experiment showed that the upcie-cuda backend matches upcie in IOPS within measurement variance, confirming that routing data through P2P DMA to GPU device memory does not measurably affect command throughput. Both preceding experiments were conducted at 512-byte I/O, where each command carries so little data that the PCIe link operates far below capacity even at tens of millions of IOPS. The bottleneck is how fast commands can be submitted and completed, not how much data the link can carry. At larger I/O sizes the bottleneck shifts: as each command carries more payload, the PCIe link itself becomes the limiting factor. The question this experiment asks is how few CPU resources and devices are required to reach that point.

This experiment fixes CPU threads at 1 and devices at 4, then sweeps I/O size across three points (512, 4096, and 8192 bytes), measuring both the payload bandwidth reported by xnvmeperf and the total PCIe receive traffic observed by DCGM at the GPU endpoint. The gap between the two is the traffic the link carries on top of the payload. A reference host-to-device bandwidth from nvbandwidth, which like xnvmeperf counts payload only, establishes the practical ceiling of the link under sustained transfers into GPU memory.

Independent Variables

The experiment fixes all parameters except NVMe command data payload size:

Variable	Value
Tool and backend	xnvmeperf + upcie-cuda
Queue depth	128
Number of CPU threads	1
Number of devices	4
I/O size	{ 512, 4096, 8192 }

Four devices were selected so that their aggregate bandwidth approaches the capacity of the GPU’s link at large I/O sizes. The Samsung PM1753 is rated at 14.5 GB/s bandwidth, and four devices therefore provide up to 4 × 14.5 GB/s = 58 GB/s aggregate, close to the 64 GB/s line rate of the PCIe Gen5 x16 link.

Metrics Collected

Metric	Reported by
Payload bandwidth (GB/s)	xnvmeperf
PCIe RX bandwidth (bytes/s)	DCGM field 1010 (PCIE_RX_BYTES)
SM activity (fraction of SMs occupied)	DCGM field 1002 (SM_ACTIVE)
GPU memory bandwidth utilization	DCGM field 1005 (DRAM_ACTIVE)
PCIe link generation and width	DCGM fields 237/238
Run validity guards	DCGM fields 100/101/112/202
Host-to-device PCIe bandwidth (GB/s)	nvbandwidth

The DCGM fields are collected as described in GPU Telemetry.

Environment

The benchmarks were run on the High-Performance Compute Server. NVMe devices are bound to user space drivers. The CPU governor is set to performance with turbo boost and SMT enabled. Each configuration is run five times and results are reported as arithmetic means.

The namespaces are formatted before the run, as described in Device Fill State.

NUMA placement was not controlled: CPUs, devices and buffers were used regardless of the node they belong to.

Execution of the Experiment

Instructions for running bench_pcie.yaml are provided in Experimental Framework.

Results

PCIe RX bandwidth by NVMe command data payload size, measured with 4 PCIe Gen5 NVMe SSDs transferring data P2P to a PCIe Gen5 GPU via the upcie-cuda backend. Each bar is stacked: the lower segment is the payload bandwidth reported by xnvmeperf; the upper segment is the remainder observed by DCGM, taken as the p95 of field 1010 rather than the mean, since the transferring samples include the ramp up. The dashed lines mark the PCIe line rate (64.0 GB/s for the Gen5 x16 link derived from the measured DCGM link fields 237/238) and the reference host-to-device bandwidth from nvbandwidth (53.3 GB/s).

Small I/O: Link Underutilized

With 512-byte payloads, xnvmeperf reports 7.9 GB/s payload bandwidth and DCGM measures 10.1 GB/s total PCIe receive traffic, corresponding to approximately 16% of the PCIe Gen5 x16 line rate. This is consistent with the IOPS-bound regime observed in the software abstraction overhead results: at small I/O sizes the constraint is NVMe command throughput, not PCIe link capacity.

Large I/O: Bandwidth Plateau

With 4096- and 8192-byte payloads, xnvmeperf reports approximately 45 GB/s of payload bandwidth in both cases, 84% of the 53.3 GB/s that nvbandwidth reaches on the same link. DCGM measures approximately 57.8 GB/s of total receive traffic, approximately 90% of the 64.0 GB/s line rate. The identical result at both I/O sizes shows that NVMe command throughput is no longer the constraint. This setup cannot separate whether the limit is the GPU’s link, the four devices, or the path between them. The plateau is reached with a single CPU thread and only four NVMe devices.

Run Validity

As expected, SM_ACTIVE (1002) is 0.0% at every I/O size, confirming that the CPU-initiated P2P path consumes no GPU compute.

Between 78% and 81% of each monitoring window qualified as transferring. DRAM_ACTIVE (1005) stays below 0.002% on average and peaks at 0.1%, so the HBM write drain is nowhere near a constraint.

The guard fields agree that the runs are comparable. The SM and memory clocks hold at 1755 MHz and 1593 MHz with the throttle reason bits clear, the replay counter stays at zero, and the link fields (237/238) report Gen5 x16 throughout, which confirms the 64.0 GB/s line rate.

PCIe Protocol Overhead

Across all three I/O sizes, the total PCIe receive bandwidth measured by DCGM exceeds the payload bandwidth reported by xnvmeperf by a consistent factor of approximately 1.28. The ratio holds across both the IOPS-bound regime at 512 bytes and the bandwidth-bound regime at 4096 and 8192 bytes, spanning a range where IOPS differ by an order of magnitude. This indicates that the excess scales with bytes transferred rather than with operation count, making the 28% overhead a stable characterization of the P2P data path regardless of operating regime.

On this path the submission and completion queues and the PRP lists reside in host memory, so command traffic does not cross the GPU’s link and is not counted in field 1010. A per-byte excess is instead consistent with PCIe packet framing: each write packet carries headers alongside a payload no larger than the link’s maximum payload size, so the cost per byte is constant.

Summary

At small I/O sizes the P2P link operates far below capacity. At 4 KiB and above, a single CPU thread driving four NVMe devices reaches a plateau of approximately 45 GB/s of payload, 84% of the 53.3 GB/s nvbandwidth reference, with total PCIe traffic at approximately 57.8 GB/s, 90% of the Gen5 x16 line rate. Total traffic exceeds payload by a consistent 28% across all tested I/O sizes, which scales with bytes transferred rather than operation count and is consistent with PCIe packet framing. Throughout, SM activity remains at zero, so the path reaches this bandwidth without consuming any GPU compute.