fio xNVMe IO Engine

Provided with xNVMe is an external fio IO engine.

Caveat

  1. For “raw” device-files only - That is, /dev/nvme0n1, /dev/nullb0 and NOT file-system files

  2. Must be used with --thread=1

Usage

Given that you have installed xNVMe then the IO engine should be available as a shared library.

The library prefix, and suffix vary based on distribution and platform, however, on a Debian-like distribution it is available as libxnvme-fio-engine.so. Location of the engine is also dist specific, but would end up in e.g. /usr/lib/.

Make sure you adjust it accordingly when using the fio-examples below.

Also, when, using the SPDK backend do:

# Unbind devices from Linux Kernel NVMe Driver
xnvme-driver

# Run fio using "be:spdk"
fio xnvme-compare-nvme.fio --section=xnvme_spdk

Aslo, when, using any of the non SPDK backends make sure that your devices are attached to the Kernel NVMe driver:

# Unbind devices from Linux Kernel NVMe Driver
xnvme-driver reset

# Run fio using "be:spdk"
fio xnvme-compare-nvme.fio --section=xnvme_spdk

See details on the parameters of the .fio file in the following section.

fio xNVMe IO Engine verification

This fio-script is used for verifying xNVMe under load.

; Verify fio IO engines with a random-write workload using 4K BS at QD16
;
; README
;
; Make sure you provide the correct path to 'libxnvme-fio-engine.so' in the sections below
;
; Take note of the "special" use of 'filename' in the sections below
;
; In the fio-file, it needs "\" to escape ":", eg.
; filename=liou\:/dev/nvme0n1
;
; On the command-line, it needs "\\" to escape ":", eg.
; --filename=liou\\:/dev/nvme0n1
;
; Job-file is intended to be used as:
;
; fio xnvme-compare.fio \
;		--section=qd1 \
;		--filename=liou:\\/dev/nvme0n1 \
;		--ioengine=external:/usr/lib/libxnvme-fio-engine.so \
;		--sqthread_poll=1
;
; or
;
; fio xnvme-compare.fio \
;		--section=qd1 \
;		--filename=/dev/nvme0n1 \
;		--ioengine=io_uring \
;		--sqthread_poll=1
;
; That is, varying the ioengine, and its options, on the command-line,
; fixing the workload, with the exception of changing io-depth using the
; --section
;
[global]
rw=randwrite
size=1G
iodepth=16
bs=4k
direct=1
verify=crc32c
thread=1

; Avoid accidentally creating device files; e.g. "/dev/nvme0n1", "/dev/nullb0"
allow_file_create=0

[default]

[qd1]
iodepth=1

[qd2]
iodepth=2

[qd4]
iodepth=4

[qd8]
iodepth=8

[qd16]
iodepth=16

[qd32]
iodepth=32

; Regular/io_uring engine

[uring]
ioengine=io_uring
hipri=0
fixedbufs=0
registerfiles=0
sqthread_poll=0
;filename=/dev/nullb0
;filename=/dev/nvme0n1

[uring_iop]
ioengine=io_uring
hipri=1
fixedbufs=0
registerfiles=1
sqthread_poll=0
;filename=/dev/nullb0
;filename=/dev/nvme0n1

[uring_sqt]
ioengine=io_uring
hipri=0
fixedbufs=0
registerfiles=1
sqthread_poll=1
;filename=/dev/nullb0
;filename=/dev/nvme0n1

[uring_all]
ioengine=io_uring
hipri=1
fixedbufs=1
registerfiles=1
sqthread_poll=1
;filename=/dev/nullb0
;filename=/dev/nvme0n1

[aio]
ioengine=libaio
;filename=/dev/nullb0
;filename=/dev/nvme0n1

[psync]
ioengine=psync
;filename=/dev/nullb0
;filename=/dev/nvme0n1

[xnvme]
ioengine=external:/usr/lib/libxnvme-fio-engine.so
;filename=liou\:/dev/nullb0?pseudo=1
;filename=liou\:/dev/nvme0n1
;filename=pci\:0000\:01\:00.0?nsid=1
;filename=pci\:0000\:03\:00.0?nsid=1

; xNVMe/io_uring engine

[xnvme_uring]
ioengine=external:/usr/lib/libxnvme-fio-engine.so
hipri=0
sqthread_poll=0
;filename=liou\:/dev/nullb0?pseudo=1
;filename=liou\:/dev/nvme0n1
;filename=/dev/nvme0n1
;be=liou

[xnvme_uring_sqt]
ioengine=external:/usr/lib/libxnvme-fio-engine.so
hipri=0
sqthread_poll=1
;filename=liou\:/dev/nullb0?pseudo=1
;filename=liou\:/dev/nvme0n1
;filename=/dev/nvme0n1
;be=liou

[xnvme_uring_iop]
ioengine=external:/usr/lib/libxnvme-fio-engine.so
hipri=1
sqthread_poll=0
;filename=liou\:/dev/nullb0?pseudo=1
;filename=liou\:/dev/nvme0n1
;filename=/dev/nvme0n1
;be=liou

[xnvme_uring_all]
ioengine=external:/usr/lib/libxnvme-fio-engine.so
hipri=1
sqthread_poll=1
;filename=liou\:/dev/nullb0?pseudo=1
;filename=liou\:/dev/nvme0n1
;filename=/dev/nvme0n1
;be=liou

[xnvme_laio]
ioengine=external:/usr/lib/libxnvme-fio-engine.so
;filename=laio\:/dev/nullb0?pseudo=1
;filename=laio\:/dev/nvme0n1
;filename=/dev/nvme0n1
;be=laio

[xnvme_spdk]
ioengine=external:/usr/lib/libxnvme-fio-engine.so
;filename=pci\:0000\:01\:00.0?nsid=1
;filename=pci\:0000\:03\:00.0?nsid=1

fio xNVMe IO Engine on NVMe Device

This fio-script can be used for comparing the performance of xNVMe to other means of shipping IO to your device. E.g. xNVMe/uring vs uring.

; Compare fio IO engines with a random-read workload using 4K BS at QD1
;
; README
;
; Make sure you provide the correct path to 'libxnvme-fio-engine.so' in the sections below
;
; Take note of the "special" use of 'filename' in the sections below
;
; In the fio-file, it needs "\" to escape ":", eg.
; filename=liou\:/dev/nvme0n1
;
; On the command-line, it needs "\\" to escape ":", eg.
; --filename=liou\\:/dev/nvme0n1
;
; Job-file is intended to be used as:
;
; fio xnvme-compare.fio \
;		--section=qd1 \
;		--filename=liou:\\/dev/nvme0n1 \
;		--ioengine=external:/usr/lib/libxnvme-fio-engine.so \
;		--sqthread_poll=1
;
; or
;
; fio xnvme-compare.fio \
;		--section=qd1 \
;		--filename=/dev/nvme0n1 \
;		--ioengine=io_uring \
;		--sqthread_poll=1
;
; That is, varying the ioengine, and its options, on the command-line,
; fixing the workload, with the exception of changing io-depth using the
; --section
;
[global]
rw=randread
size=12G
iodepth=1
bs=4K
direct=1
thread=1
time_based=1
runtime=7
ramp_time=3
norandommap=1

; Avoid accidentally creating device files; e.g. "/dev/nvme0n1", "/dev/nullb0"
allow_file_create=0

[default]

[qd1]
iodepth=1

[qd2]
iodepth=2

[qd4]
iodepth=4

[qd8]
iodepth=8

[qd16]
iodepth=16

[qd32]
iodepth=32

; Regular/io_uring engine

[uring]
ioengine=io_uring
hipri=0
fixedbufs=0
registerfiles=0
sqthread_poll=0
;filename=/dev/nullb0
;filename=/dev/nvme0n1

[uring_iop]
ioengine=io_uring
hipri=1
fixedbufs=0
registerfiles=1
sqthread_poll=0
;filename=/dev/nullb0
;filename=/dev/nvme0n1

[uring_sqt]
ioengine=io_uring
hipri=0
fixedbufs=0
registerfiles=1
sqthread_poll=1
;filename=/dev/nullb0
;filename=/dev/nvme0n1

[uring_all]
ioengine=io_uring
hipri=1
fixedbufs=1
registerfiles=1
sqthread_poll=1
;filename=/dev/nullb0
;filename=/dev/nvme0n1

[aio]
ioengine=libaio
;filename=/dev/nullb0
;filename=/dev/nvme0n1

[psync]
ioengine=psync
;filename=/dev/nullb0
;filename=/dev/nvme0n1

[xnvme]
ioengine=external:/usr/lib/libxnvme-fio-engine.so
;filename=liou\:/dev/nullb0?pseudo=1
;filename=liou\:/dev/nvme0n1
;filename=pci\:0000\:01\:00.0?nsid=1
;filename=pci\:0000\:03\:00.0?nsid=1

; xNVMe/io_uring engine

[xnvme_uring]
ioengine=external:/usr/lib/libxnvme-fio-engine.so
hipri=0
sqthread_poll=0
;filename=liou\:/dev/nullb0?pseudo=1
;filename=liou\:/dev/nvme0n1
;filename=/dev/nvme0n1
;be=liou

[xnvme_uring_sqt]
ioengine=external:/usr/lib/libxnvme-fio-engine.so
hipri=0
sqthread_poll=1
;filename=liou\:/dev/nullb0?pseudo=1
;filename=liou\:/dev/nvme0n1
;filename=/dev/nvme0n1
;be=liou

[xnvme_uring_iop]
ioengine=external:/usr/lib/libxnvme-fio-engine.so
hipri=1
sqthread_poll=0
;filename=liou\:/dev/nullb0?pseudo=1
;filename=liou\:/dev/nvme0n1
;filename=/dev/nvme0n1
;be=liou

[xnvme_uring_all]
ioengine=external:/usr/lib/fio-engine/libxnvme-fio-engine.so
hipri=1
sqthread_poll=1
;filename=liou\:/dev/nullb0?pseudo=1
;filename=liou\:/dev/nvme0n1
;filename=/dev/nvme0n1
;be=liou

[xnvme_laio]
ioengine=external:/usr/lib/libxnvme-fio-engine.so
;filename=laio\:/dev/nullb0?pseudo=1
;filename=laio\:/dev/nvme0n1
;filename=/dev/nvme0n1
;be=laio

[xnvme_spdk]
ioengine=external:/usr/lib/libxnvme-fio-engine.so
;filename=pci\:0000\:01\:00.0?nsid=1
;filename=pci\:0000\:03\:00.0?nsid=1

fio xNVMe IO Engine on NVMe Device with Zoned Command Set

This fio-script provides the basics for running on an NVMe device with the Zoned Command Set enabled.

; Running xNVMe/fio on a Zoned Device
;
; README
;
; Make sure you provide the correct path to 'libxnvme-fio-engine.so' in the
; sections below
;
; Take note of the "special" use of 'filename' in the sections below
;
; In the fio-file, it needs "\" to escape ":", eg.
; filename=liou\:/dev/nvme0n2
;
; On the command-line, it needs "\\" to escape ":", eg.
; --filename=liou\\:/dev/nvme0n2
;
; Job-file is intended to be used as:
;
;   fio xnvme-zoned.fio \
;     --section=verify \
;     --ioengine=external:/usr/lib/libxnvme-fio-engine.so \
;     --filename=liou:\\/dev/nvme0n2
;
; Have a look at xnvme-compare.fio, for the details on instrumenting the
; different backends and comparing performance.
;
; I/O errors have been observed with fio 3.20, however, not with 3.22, in any
; case, if you encounter them, then you can clean up the drive by first
; invoking:
;
;   zoned mgmt-reset /dev/nvme0n2 --slba 0x0 --all
;
; Which will prepare the drive
;
[global]
;ioengine=external:/usr/lib/libxnvme-fio-engine.so
zonemode=zbd
size=1G
iodepth=1
bs=4K
direct=1
thread=1
;time_based=1
;runtime=10
ramp_time=1
norandommap=1
; If fio complains about zone-size, then run:
; zoned info /dev/nvme0n1 it will provide the values you need, then define it:
; zonesize=nsect * nbytes
;
;zonesize=

; Avoid accidentally creating device files; e.g. "/dev/nvme0n1", "/dev/nullb0"
allow_file_create=0

[default]

[write]
rw=write

[read]
rw=read

[verify]
rw=write
verify=crc32c

Build Notes

The fio xNVMe IO engine is enabled by default via the autoconfiguration produced by:

make
sudo make install-deb

If, for some reason you are manually instrumenting CMake / configuration, then the options to enable/disable the IO engine are:

--enable-tools-fio
--disable-tools-fio