xnvme#
xnvme is the entry point to the xNVMe command line. Run it without
arguments for the commands to start with and the groups the rest are in:
xnvme
Usage: xnvme <command> [<args>] [--help]
Find NVMe devices, inspect and benchmark them, and manage userspace drivers
Start here:
xnvme list NVMe controller functions and their driver
xnvme info <uri> Retrieve derived information for given device
xnvme enum Enumerate devices on the system
xnvme perf Run IO benchmarks against xNVMe devices
Set up the host:
xnvme env Relevant system config and library introspection
xnvme driver Bind the driver a backend needs, or show them
Serve devices:
xnvme homi Share NVMe controllers between processes
xnvme qublk Serve an xNVMe device as a Linux block device
Command passthru:
xnvme cmd-admin An admin command, as 'nvme admin-passthru' takes it
xnvme cmd-io An I/O command, as 'nvme io-passthru' takes it
xnvme cmd-iov An I/O command with its data in --vec-cnt segments
Groups, run one to list its commands:
xnvme test installed test programs
xnvme example installed example programs
Run a group to list its commands, and append --help to any command for its
arguments. Commands such as perf, homi and qublk
are separate programs, which xnvme looks up on its exec path,
the install directory unless XNVME_EXEC_PATH names others;
xnvme env library shows it as exec_path, and xnvme --exec-path
prints it alone, for scripts.
A project outside xNVMe adds a command as an extension. It installs its
program xnvme-<name> into the directory that pkg-config
--variable=libexecdir xnvme prints, and xnvme <name> runs it. To be
listed in the overview, it also installs <name>.extension into the
directory that pkg-config --variable=extensiondir xnvme prints:
section=serve
description=Serve an xNVMe device over a protocol of its own
section is start, host, serve or passthru, and lines
starting with # are comments. A test or an example installs as
xnvme-test-<name> or xnvme-example-<name>, and xnvme test or
xnvme example lists it without a .extension. A program named after a
command of xnvme itself is never run.
As xnvme often runs as root, it runs a program, and reads a
.extension, only when the file and its directory belong to root or to the
user running xnvme, under sudo the user who ran it, and only they and
their groups can write them.
xnvme env says, as YAML, whether the host is ready for the userspace
drivers: the IOMMU groups, whether each kernel module is available, built in
and loaded, the hugepages of each size and NUMA node, where hugetlbfs is
mounted, memlock, the device nodes, the NVMe controllers by driver, and the
queues of the kernel nvme driver, each with a fix where something is
missing. For the kernel driver that is poll_queues: polled I/O through
io_uring needs it above zero, which takes a module option and a reload of
nvme, and a reload drops every NVMe device on the host, so xnvme says
how rather than doing it. It then prints the library: the
version, the build configuration and the backends. xnvme env system and
xnvme env library print one of the two. It changes nothing.
The fixes name the hugepages and iommu tools, installed with
pipx install hugepages iommu.
xnvme list shows the NVMe controller functions on the PCI bus, found with
xnvme_scan(), as YAML: the driver each is on and its IOMMU group, read from
sysfs without opening anything. xnvme driver --be <backend> <bdf> moves a
controller to the driver the backend needs, and prints the outcome as YAML,
each controller with the driver it is on, whether that changed, and why not.
A backend driving the controller itself, spdk, libvfn or upcie,
needs a userspace driver: on Linux vfio-pci, or uio_pci_generic
without an IOMMU, and vfio-pci alone for libvfn; on FreeBSD,
nic_uio. Every other backend needs the kernel nvme driver, so
--be io_uring returns a controller to it. Only the named controller is
rebound, and --all takes every one not in use. Bare xnvme driver
prints xnvme list and then xnvme env. One whose
namespaces are mounted, held by device-mapper, md or LVM, used as swap or open
in a process is refused, as is one sharing its IOMMU group with a device on
another driver; one whose namespace carries a filesystem or partition table is
refused unless --force. The binding is done by the platform, through
xnvme_driver_for_be(), xnvme_driver_attach() and
xnvme_driver_detach() in libxnvme, and is not
supported on macOS and Windows. Nothing is recorded on the side: what is
shown is what sysfs says, and whether a controller is in use is checked when
it is to move, which --dry-run does without moving it.
xnvme cmd-admin, cmd-io and cmd-iov send a raw command, and take
the options of nvme admin-passthru and nvme io-passthru, the same long
names and the same letters, so a command line moves between the two by
changing the program and the subcommand. The device is any xNVMe URI, so
the same command reaches a controller on a userspace driver, which
nvme-cli cannot:
nvme admin-passthru /dev/nvme0 --opcode=0x06 --cdw10=1 --data-len=4096 -r
xnvme cmd-admin /dev/nvme0n1 --opcode=0x06 --cdw10=1 --data-len=4096 -r
xnvme cmd-admin 0000:03:00.0 --opcode=0x06 --cdw10=1 --data-len=4096 -r --be upcie
The options choosing the path, --be, --admin, --sync, --async,
--dev-nsid and --homi-id, are long options only. cmd-iov sends the
data in --vec-cnt equal segments. --timeout and --output-format json
are refused rather than ignored.
The admin commands with their output decoded, Identify, the log pages,
features, format, sanitize, FDP and the resets, are xnvme test adm, which
the tests run through every backend. For a controller on the kernel driver,
nvme-cli is the tool for these; xnvme cmd-admin sends any of them raw,
on any backend.
The installed test programs are listed and run the same way:
xnvme test
The test programs:
xnvme test adm
xnvme test async-intf
xnvme test async-limbo
xnvme test be
xnvme test bits
xnvme test buf
xnvme test cli
xnvme test compare
xnvme test compat
xnvme test copy
xnvme test ctrlr
xnvme test cuse-admin64-identify
xnvme test dd
xnvme test delay-identification
xnvme test dual-be
xnvme test enum
xnvme test file
xnvme test ioworker
xnvme test kv
xnvme test kvs
xnvme test lblk
xnvme test liburing
xnvme test map
xnvme test multi-ctrlr
xnvme test nvm
xnvme test scc
xnvme test spec
xnvme test threads
xnvme test xnvme-cli
xnvme test xnvme-file
xnvme test znd-append
xnvme test znd-explicit-open
xnvme test znd-state
xnvme test znd-zrwa
xnvme test zns
See 'xnvme test <name> --help' for the usage of one
xnvme --help prints the same overview.
The bash completion, installed with xnvme, asks it what to complete:
xnvme --list-cmds prints the commands and the installed programs,
xnvme --list-cmds=<group> those of a group such as test, and each
program prints its subcommands with --list-cmds. It therefore completes what
is installed, and the man pages, man xnvme and man xnvme-<command>, are
generated from the same --help output shown here.