fio xNVMe IO Engine¶
Provided with xNVMe is an external fio IO engine.
Caveat¶
For “raw” device-files only - That is,
/dev/nvme0n1
,/dev/nullb0
and NOT file-system filesMust be used with
--thread=1
Usage¶
Given that you have installed xNVMe then the IO engine should be available as a shared library.
The library prefix, and suffix vary based on distribution and platform,
however, on a Debian-like distribution it is available under the filename
libxnvme-fio-engine.so
and located in /usr/lib/
when installed.
In the examples below, it is assumed that the io-engine is available at:
--ioengine=external:/usr/lib/libxnvme-fio-engine.so
Adjust the fio --ioengine=external:
to match your system.
Also, when, using the SPDK backend do:
# Unbind devices from Linux Kernel NVMe Driver
xnvme-driver
# Run fio using "be:spdk"
fio xnvme-compare-nvme.fio --section=xnvme_spdk
Aslo, when, using any of the non SPDK backends make sure that your devices are attached to the Kernel NVMe driver:
# Unbind devices from Linux Kernel NVMe Driver
xnvme-driver reset
# Run fio using "be:spdk"
fio xnvme-compare-nvme.fio --section=xnvme_spdk
See details on the parameters of the .fio
file in the following section.
fio xNVMe IO Engine verification¶
This fio-script is used for verifying xNVMe under load.
; Verify fio IO engines with a random-write workload using BS=4k at QD=16
;
; Writes, at random, 1GB at QD16 using 4K BS and verifies it.
;
; README
;
; Make sure you provide the correct path to 'libxnvme-fio-engine.so' when
; invoking fio. This job-file is intended to be used as:
;
; # Use the built-in io_uring engine to get baseline numbers
; fio xnvme-compare.fio \
; --section=default \
; --ioengine=io_uring \
; --sqthread_poll=1 \
; --filename=/dev/nvme0n1
;
; # Use the xNVMe io-engine engine with Linux backend and libaio async. impl.
; fio xnvme-compare.fio \
; --section=default \
; --ioengine=external:/usr/lib/libxnvme-fio-engine.so \
; --sqthread_poll=1 \
; --async=aio \
; --filename=/dev/nvme0n1
;
; # Use the xNVMe io-engine engine with Linux backend and io_uring async. impl.
; fio xnvme-compare.fio \
; --section=default \
; --ioengine=external:/usr/lib/libxnvme-fio-engine.so \
; --sqthread_poll=1 \
; --async=iou \
; --filename=/dev/nvme0n1
;
; # Use the xNVMe io-engine engine with SPDK backend
; fio xnvme-compare.fio \
; --ioengine=external:/usr/lib/libxnvme-fio-engine.so \
; --section=default \
; --filename=pci\\:0000\\:01\\:00.0?nsid=1
;
; NOTE: The URI encoded in the filename above, the ":" must be escaped.
;
; On the command-line using two "\\":
;
; --filename=pci\\:0000\\:01\\:00.0?nsid=1
;
; Within a fio-script using a single "\":
;
; filename=pci\:0000\:01\:00.0?nsid=1
;
; NOTE: If you want to override the default bs, iodepth, and workload, then
; invoke it as:
;
; FIO_BS="512" FIO_RW="verify" FIO_IODEPTH=16 fio xnvme-compare.fio \
; --section=override
;
[global]
rw=randwrite
size=1G
iodepth=16
bs=4k
direct=1
thread=1
verify=crc32c
; Avoid accidentally creating device files; e.g. "/dev/nvme0n1", "/dev/nullb0"
allow_file_create=0
[default]
[override]
rw=${FIO_RW}
iodepth=${FIO_IODEPTH}
bs=${FIO_BS}
fio xNVMe IO Engine on NVMe Device¶
This fio-script can be used for comparing the performance of xNVMe to other means of shipping IO to your device. E.g. xNVMe/uring vs uring.
; Compare fio IO engines with a random-read workload using BS=4k at QD=1
;
; README
;
; Make sure you provide the correct path to 'libxnvme-fio-engine.so' when
; invoking fio. This job-file is intended to be used as:
;
; # Use the built-in io_uring engine to get baseline numbers
; fio xnvme-compare.fio \
; --section=default \
; --ioengine=io_uring \
; --sqthread_poll=1 \
; --filename=/dev/nvme0n1
;
; # Use the xNVMe io-engine engine with Linux backend and libaio async. impl.
; fio xnvme-compare.fio \
; --section=default \
; --ioengine=external:/usr/lib/libxnvme-fio-engine.so \
; --sqthread_poll=1 \
; --async=aio \
; --filename=/dev/nvme0n1
;
; # Use the xNVMe io-engine engine with Linux backend and io_uring async. impl.
; fio xnvme-compare.fio \
; --section=default \
; --ioengine=external:/usr/lib/libxnvme-fio-engine.so \
; --sqthread_poll=1 \
; --async=iou \
; --filename=/dev/nvme0n1
;
; # Use the xNVMe io-engine engine with SPDK backend
; fio xnvme-compare.fio \
; --ioengine=external:/usr/lib/libxnvme-fio-engine.so \
; --section=default \
; --filename=pci\\:0000\\:01\\:00.0?nsid=1
;
; NOTE: The URI encoded in the filename above, the ":" must be escaped.
;
; On the command-line using two "\\":
;
; --filename=pci\\:0000\\:01\\:00.0?nsid=1
;
; Within a fio-script using a single "\":
;
; filename=pci\:0000\:01\:00.0?nsid=1
;
; NOTE: If you want to override the default bs, iodepth, and workload, then
; invoke it as:
;
; FIO_BS="512" FIO_RW="verify" FIO_IODEPTH=16 fio xnvme-compare.fio \
; --section=override
;
[global]
rw=randread
size=12G
iodepth=1
bs=4K
direct=1
thread=1
time_based=1
runtime=7
ramp_time=3
norandommap=1
; Avoid accidentally creating device files; e.g. "/dev/nvme0n1", "/dev/nullb0"
allow_file_create=0
[default]
[override]
rw=${FIO_RW}
iodepth=${FIO_IODEPTH}
bs=${FIO_BS}
fio xNVMe IO Engine on NVMe Device with Zoned Command Set¶
This fio-script provides the basics for running on an NVMe device with the Zoned Command Set enabled.
; Running xNVMe/fio on a Zoned Device
;
; Writes 1GB at QD1 using 4K BS and verifies it.
;
; README
;
; Make sure you provide the correct path to 'libxnvme-fio-engine.so' when
; invoking fio. This job-file is intended to be used as:
;
; # Use the built-in io_uring engine to get baseline numbers
; fio xnvme-compare.fio \
; --section=default \
; --ioengine=io_uring \
; --sqthread_poll=1 \
; --filename=/dev/nvme0n1
;
; # Use the xNVMe io-engine engine with Linux backend and libaio async. impl.
; fio xnvme-compare.fio \
; --section=default \
; --ioengine=external:/usr/lib/libxnvme-fio-engine.so \
; --sqthread_poll=1 \
; --async=aio \
; --filename=/dev/nvme0n1
;
; # Use the xNVMe io-engine engine with Linux backend and io_uring async. impl.
; fio xnvme-compare.fio \
; --section=default \
; --ioengine=external:/usr/lib/libxnvme-fio-engine.so \
; --sqthread_poll=1 \
; --async=iou \
; --filename=/dev/nvme0n1
;
; # Use the xNVMe io-engine engine with SPDK backend
; fio xnvme-compare.fio \
; --ioengine=external:/usr/lib/libxnvme-fio-engine.so \
; --section=default \
; --filename=pci\\:0000\\:01\\:00.0?nsid=1
;
; NOTE: The URI encoded in the filename above, the ":" must be escaped.
;
; On the command-line using two "\\":
;
; --filename=pci\\:0000\\:01\\:00.0?nsid=1
;
; Within a fio-script using a single "\":
;
; filename=pci\:0000\:01\:00.0?nsid=1
;
; NOTE: If you want to override the default bs, iodepth, and workload, then
; invoke it as:
;
; FIO_BS="512" FIO_RW="verify" FIO_IODEPTH=16 fio xnvme-compare.fio \
; --section=override
;
; NOTE: I/O errors have been observed with fio 3.20, however, not with 3.22,
; in any case, if you encounter them, then you can clean up the drive by first
; invoking:
;
; # zoned mgmt-reset /dev/nvme0n2 --slba 0x0 --all
;
; This will reset all zones on the device to EMPTY aka. wipe the entire device.
;
[global]
zonemode=zbd
rw=write
size=1G
iodepth=1
bs=4K
direct=1
thread=1
ramp_time=1
norandommap=1
verify=crc32c
; Avoid accidentally creating device files; e.g. "/dev/nvme0n1", "/dev/nullb0"
allow_file_create=0
;
; NOTE: If fio complains about zone-size, then run:
;
; # zoned info /dev/nvme0n1
;
; The command will provide the values you need, then in the fio-script define:
;
; zonesize=nsect * nbytes
;
;zonesize=
[default]
[override]
rw=${FIO_RW}
iodepth=${FIO_IODEPTH}
bs=${FIO_BS}
Build Notes¶
The fio
xNVMe IO engine is enabled by default via the autoconfiguration
produced by:
make
sudo make install-deb
If, for some reason you are manually instrumenting CMake / configuration, then the options to enable/disable the IO engine are:
--enable-tools-fio
--disable-tools-fio