fio xNVMe IO Engine

Provided with xNVMe is an external fio IO engine.

Caveat

1. For “raw” device-files only - That is, /dev/nvme0n1, /dev/nullb0 and NOT file-system files

2. Must be used with --thread=1

Usage

Given that you have installed xNVMe then the IO engine should be available as a shared library.

The library prefix, and suffix vary based on distribution and platform, however, on a Debian-like distribution it is available as libxnvme-fio-engine.so. Location of the engine is also dist specific, but would end up in e.g. /usr/lib/.

Make sure you adjust it accordingly when using the fio-examples below.

Also, when, using the SPDK backend do:

# Unbind devices from Linux Kernel NVMe Driver
xnvme-driver

# Run fio using "be:spdk"
fio xnvme-compare-nvme.fio --section=xnvme_spdk

Aslo, when, using any of the non SPDK backends make sure that your devices are attached to the Kernel NVMe driver:

# Unbind devices from Linux Kernel NVMe Driver
xnvme-driver reset

# Run fio using "be:spdk"
fio xnvme-compare-nvme.fio --section=xnvme_spdk

See details on the parameters of the .fio file in the following section.

fio xNVMe IO Engine verification

This fio-script is used for verifying xNVMe under load.

; Verify fio IO engines with a random-write workload using BS=4k at QD=16
;
; Writes, at random, 1GB at QD16 using 4K BS and verifies it.
;
; README
;
; Make sure you provide the correct path to 'libxnvme-fio-engine.so' when
; invoking fio. This job-file is intended to be used as:
;
; # Use the built-in io_uring engine to get baseline numbers
; fio xnvme-compare.fio \
;   --section=default \
;   --ioengine=io_uring \
;   --sqthread_poll=1 \
;   --filename=/dev/nvme0n1
;
; # Use the xNVMe io-engine engine with Linux backend and libaio async. impl.
; fio xnvme-compare.fio \
;   --section=default \
;   --ioengine=external:/usr/lib/libxnvme-fio-engine.so \
;   --sqthread_poll=1 \
;   --async=aio \
;   --filename=/dev/nvme0n1
;
; # Use the xNVMe io-engine engine with Linux backend and io_uring async. impl.
; fio xnvme-compare.fio \
;   --section=default \
;   --ioengine=external:/usr/lib/libxnvme-fio-engine.so \
;   --sqthread_poll=1 \
;   --async=iou \
;   --filename=/dev/nvme0n1
;
; # Use the xNVMe io-engine engine with SPDK backend
; fio xnvme-compare.fio \
;   --ioengine=external:/usr/lib/libxnvme-fio-engine.so \
;   --section=default \
;   --filename=pci\\:0000\\:01\\:00.0?nsid=1
;
; NOTE: The URI encoded in the filename above, the ":" must be escaped.
;
; On the command-line using two "\\":
;
; --filename=pci\\:0000\\:01\\:00.0?nsid=1
;
; Within a fio-script using a single "\":
;
; filename=pci\:0000\:01\:00.0?nsid=1
;
; NOTE: If you want to override the default bs, iodepth, and workload, then
; invoke it as:
;
; FIO_BS="512" FIO_RW="verify" FIO_IODEPTH=16 fio xnvme-compare.fio \
;   --section=override
;
[global]
rw=randwrite
size=1G
iodepth=16
bs=4k
direct=1
thread=1
verify=crc32c

; Avoid accidentally creating device files; e.g. "/dev/nvme0n1", "/dev/nullb0"
allow_file_create=0

[default]

[override]
rw=${FIO_RW}
iodepth=${FIO_IODEPTH}
bs=${FIO_BS}

fio xNVMe IO Engine on NVMe Device

This fio-script can be used for comparing the performance of xNVMe to other means of shipping IO to your device. E.g. xNVMe/uring vs uring.

; Compare fio IO engines with a random-read workload using BS=4k at QD=1
;
; README
;
; Make sure you provide the correct path to 'libxnvme-fio-engine.so' when
; invoking fio. This job-file is intended to be used as:
;
; # Use the built-in io_uring engine to get baseline numbers
; fio xnvme-compare.fio \
;   --section=default \
;   --ioengine=io_uring \
;   --sqthread_poll=1 \
;   --filename=/dev/nvme0n1
;
; # Use the xNVMe io-engine engine with Linux backend and libaio async. impl.
; fio xnvme-compare.fio \
;   --section=default \
;   --ioengine=external:/usr/lib/libxnvme-fio-engine.so \
;   --sqthread_poll=1 \
;   --async=aio \
;   --filename=/dev/nvme0n1
;
; # Use the xNVMe io-engine engine with Linux backend and io_uring async. impl.
; fio xnvme-compare.fio \
;   --section=default \
;   --ioengine=external:/usr/lib/libxnvme-fio-engine.so \
;   --sqthread_poll=1 \
;   --async=iou \
;   --filename=/dev/nvme0n1
;
; # Use the xNVMe io-engine engine with SPDK backend
; fio xnvme-compare.fio \
;   --ioengine=external:/usr/lib/libxnvme-fio-engine.so \
;   --section=default \
;   --filename=pci\\:0000\\:01\\:00.0?nsid=1
;
; NOTE: The URI encoded in the filename above, the ":" must be escaped.
;
; On the command-line using two "\\":
;
; --filename=pci\\:0000\\:01\\:00.0?nsid=1
;
; Within a fio-script using a single "\":
;
; filename=pci\:0000\:01\:00.0?nsid=1
;
; NOTE: If you want to override the default bs, iodepth, and workload, then
; invoke it as:
;
; FIO_BS="512" FIO_RW="verify" FIO_IODEPTH=16 fio xnvme-compare.fio \
;   --section=override
;
[global]
rw=randread
size=12G
iodepth=1
bs=4K
direct=1
thread=1
time_based=1
runtime=7
ramp_time=3
norandommap=1

; Avoid accidentally creating device files; e.g. "/dev/nvme0n1", "/dev/nullb0"
allow_file_create=0

[default]

[override]
rw=${FIO_RW}
iodepth=${FIO_IODEPTH}
bs=${FIO_BS}

fio xNVMe IO Engine on NVMe Device with Zoned Command Set

This fio-script provides the basics for running on an NVMe device with the Zoned Command Set enabled.

; Running xNVMe/fio on a Zoned Device
;
; Writes 1GB at QD1 using 4K BS and verifies it.
;
; README
;
; Make sure you provide the correct path to 'libxnvme-fio-engine.so' when
; invoking fio. This job-file is intended to be used as:
;
; # Use the built-in io_uring engine to get baseline numbers
; fio xnvme-compare.fio \
;   --section=default \
;   --ioengine=io_uring \
;   --sqthread_poll=1 \
;   --filename=/dev/nvme0n1
;
; # Use the xNVMe io-engine engine with Linux backend and libaio async. impl.
; fio xnvme-compare.fio \
;   --section=default \
;   --ioengine=external:/usr/lib/libxnvme-fio-engine.so \
;   --sqthread_poll=1 \
;   --async=aio \
;   --filename=/dev/nvme0n1
;
; # Use the xNVMe io-engine engine with Linux backend and io_uring async. impl.
; fio xnvme-compare.fio \
;   --section=default \
;   --ioengine=external:/usr/lib/libxnvme-fio-engine.so \
;   --sqthread_poll=1 \
;   --async=iou \
;   --filename=/dev/nvme0n1
;
; # Use the xNVMe io-engine engine with SPDK backend
; fio xnvme-compare.fio \
;   --ioengine=external:/usr/lib/libxnvme-fio-engine.so \
;   --section=default \
;   --filename=pci\\:0000\\:01\\:00.0?nsid=1
;
; NOTE: The URI encoded in the filename above, the ":" must be escaped.
;
; On the command-line using two "\\":
;
; --filename=pci\\:0000\\:01\\:00.0?nsid=1
;
; Within a fio-script using a single "\":
;
; filename=pci\:0000\:01\:00.0?nsid=1
;
; NOTE: If you want to override the default bs, iodepth, and workload, then
; invoke it as:
;
; FIO_BS="512" FIO_RW="verify" FIO_IODEPTH=16 fio xnvme-compare.fio \
;   --section=override
;
; NOTE: I/O errors have been observed with fio 3.20, however, not with 3.22,
; in any case, if you encounter them, then you can clean up the drive by first
; invoking:
;
; # zoned mgmt-reset /dev/nvme0n2 --slba 0x0 --all
;
; This will reset all zones on the device to EMPTY aka. wipe the entire device.
;
[global]
zonemode=zbd
rw=write
size=1G
iodepth=1
bs=4K
direct=1
thread=1
ramp_time=1
norandommap=1
verify=crc32c
; Avoid accidentally creating device files; e.g. "/dev/nvme0n1", "/dev/nullb0"
allow_file_create=0
;
; NOTE: If fio complains about zone-size, then run:
;
; # zoned info /dev/nvme0n1
;
; The command will provide the values you need, then in the fio-script define:
;
; zonesize=nsect * nbytes
;
;zonesize=

[default]

[override]
rw=${FIO_RW}
iodepth=${FIO_IODEPTH}
bs=${FIO_BS}

Build Notes

The fio xNVMe IO engine is enabled by default via the autoconfiguration produced by:

make
sudo make install-deb

If, for some reason you are manually instrumenting CMake / configuration, then the options to enable/disable the IO engine are:

--enable-tools-fio
--disable-tools-fio