qublk#
qublk is a ublk server backed by xNVMe. It creates a /dev/ublkbN
block device, receives READ / WRITE / FLUSH requests, and on a
backend passing NVMe commands through also DISCARD / WRITE_ZEROES, from
the Linux ublk_drv over io_uring, and services them through the
xNVMe asynchronous command interface. Thus, any xNVMe backend, such as
uPCIe or io_uring, becomes usable as an ordinary block
device.
REQ_FUA and REQ_PREFLUSH are supported, as are multiple ublk hardware
queues via --nqueues. Discard and write zeroes are described below.
Usage: xnvme qublk <command> [<args>]
Expose an xNVMe device as a ublk block-device
Where <command> is one of:
serve | Serve a ublk block-device for each of the given xNVMe devices
del | Delete a ublk device left behind by a killed server
See 'xnvme qublk <command> --help' for the description of [<args>]
See 'xnvme qublk --version' for the library version and the revision it was built from
qublk - ublk server backed by xNVMe -- ver: {major: 0, minor: 7, patch: 5, vcs: 'e6cf2f4'}
Requirements#
qublk is Linux-only and is built only when liburing and
<linux/ublk_cmd.h> are available. At runtime it requires:
rootprivilegesthe ublk driver, loaded with
modprobe ublk_drvfor user space backends such as uPCIe, a device bound to
uio_pci_genericand hugepages configured; see uPCIe
The Linux-NVMe-driver ioctl() mimic (see below) additionally needs a kernel
new enough to carry NVME_IOCTL_IO64_CMD_VEC in <linux/nvme_ioctl.h>
(5.17+) alongside <linux/fuse.h>, both at build time;
-Dwith-cuse=disabled skips it explicitly, and without either header
qublk still builds, quietly serving every device without its -nvme
character device, since that is a permanent build-time condition rather
than a one-off failure; a CUSE device that fails to come up at runtime for
some other reason still warns and does the same.
serve — Serve a block-device#
Opens the given device URIs, adds a ublk device for each, and serves them
until SIGINT / SIGTERM, upon which it performs a clean teardown
(STOP_DEV followed by DEL_DEV).
When --dev-id is not given, the kernel assigns the device identifiers;
when given, the devices are numbered consecutively from it.
When --max-io-bytes is not given, the per-IO buffer size defaults to the
smaller of 1MiB and the controller MDTS.
With --user-copy the server moves the request data itself, with
pread() / pwrite() on /dev/ublkc<N> (UBLK_F_USER_COPY, kernel
6.5+), instead of the kernel reading and writing the server’s buffers.
Zoned namespaces always use it.
By default, each queue is served by a thread of its own. With --cpumask or
--cpulist, a thread is pinned to each given CPU and the queues are spread
evenly across them.
Usage: xnvme qublk serve [<uri>...] [<args>]
Serve a ublk block-device for each of the given xNVMe devices
Positional arguments:
[uri ...] ; Device URI e.g. '/dev/nvme0n1', '0000:01:00.1', '10.9.8.1:8888', '\\.\PhysicalDrive1'
Where <args> include:
[ --qdepth NUM ] ; Use given 'NUM' as queue max capacity
[ --nqueues NUM ] ; Number of queues per device
[ --dev-id NUM ] ; Use given 'NUM' as device identifier
[ --max-io-bytes NUM ] ; Use given 'NUM' as per-IO buffer size in bytes
[ --cpumask STRING ] ; Hex CPU bitmask for thread pinning (e.g. 0x3)
[ --cpulist STRING ] ; Comma-separated list of CPU ids or CPU id ranges for thread pinning (e.g. 0-3,4,6)
[ --no-cuse ] ; Disable the NVMe-driver ioctl() CUSE mimic
[ --user-copy ] ; Move ublk request data with pread()/pwrite() on /dev/ublkc<N> (UBLK_F_USER_COPY)
With <args> for backend:
[ --be STRING ] ; xNVMe backend, e.g. 'spdk', 'libvfn', 'upcie'
[ --homi-id NUM ] ; The HOMI identifier to share controllers under
[ --help ] ; Show usage / help
See 'xnvme qublk --help' for other commands
qublk - ublk server backed by xNVMe -- ver: {major: 0, minor: 7, patch: 5, vcs: 'e6cf2f4'}
Example — NVMe block device via io_uring:
xnvme qublk serve /dev/nvme0n1 --be io_uring --qdepth 64
Example — user space NVMe via uPCIe, with multiple hardware queues:
xnvme qublk serve 0000:01:00.0 --be upcie --qdepth 64 --nqueues 4
Example — four devices via uPCIe, served by two CPUs:
xnvme qublk serve 0000:01:00.0 0000:02:00.0 0000:03:00.0 0000:04:00.0 \
--be upcie --qdepth 64 --cpulist 0-1
While qublk is running, each device appears as /dev/ublkb<N>.
A device driven by one of xNVMe’s user-space NVMe drivers (--be spdk,
libvfn or upcie) also gets a /dev/ublkb<N>-nvme character device
answering the Linux kernel NVMe driver’s ioctl() interface (NVME_IOCTL_ID,
NVME_IOCTL_ADMIN_CMD, NVME_IOCTL_IO_CMD, NVME_IOCTL_ADMIN64_CMD,
NVME_IOCTL_IO64_CMD, NVME_IOCTL_IO64_CMD_VEC), so tools such as
nvme-cli address it like a kernel-attached namespace. Other devices get
none: a kernel-managed NVMe namespace already has /dev/nvme<X>n<Y> and
/dev/ng<X>n<Y>, and a device that is not NVMe, such as /dev/sda or a
file, has no NVMe interface to mimic. Pass --no-cuse to skip it; a failed
CUSE device only disables itself, the block device still comes up.
Passthru data and metadata, together with the
nvme_passthru_cmd/nvme_passthru_cmd64 struct itself, are capped at
128KiB by the kernel’s CUSE ioctl handling, not the device’s real
MDTS. NVME_IOCTL_IO64_CMD_VEC additionally rejects a request whose
vec_cnt is above 125: every retry round resends the same iovec list as
both the fetch and the scatter-back list, and the kernel’s own
FUSE_IOCTL_MAX_IOV (256) bounds their combined count, well below the
real driver’s UIO_MAXIOV.
Discard and write zeroes#
DISCARD is served as one Dataset Management deallocate command with a
single range, and WRITE_ZEROES as one Write Zeroes command. Each is
advertised in /sys/block/ublkb<N>/queue/discard_max_bytes and
write_zeroes_max_bytes only when all of these hold; otherwise the value is
0 and the kernel keeps the op away from qublk:
the backend passes NVMe commands through:
io_uring_cmd,spdk,libvfn,upcie, oremu/thrpoolover thenvme,spdk,libvfnorupciesync path.the controller lists the command in
ONCS.the namespace is not zoned.
The limits follow the kernel NVMe driver. Discard uses DMRSL when it fits
the 32-bit sector count, else UINT32_MAX sectors rounded down to whole
blocks. Write zeroes uses the transfer cap xNVMe derives from MDTS, or
WZSL when reported, and at most 65536 blocks. WZSL is counted in 4KiB
pages, since xNVMe does not expose CAP.MPSMIN. --max-io-bytes does
not affect either limit. The startup line prints both limits in sectors.
A command refused with Invalid Command Opcode or Invalid Field in Command
completes as EOPNOTSUPP, as with the kernel driver; any other error is
EIO. Write Zeroes sets the deallocate bit when DLFEAT reports that
deallocated blocks read as zeroes and that Write Zeroes may deallocate, unless
the request carries UBLK_IO_F_NOUNMAP.
Zoned namespaces#
A ZNS namespace is served as a host-managed zoned block device, so blkzone,
zonefs, zoned btrfs and fio --zonemode=zbd work on
/dev/ublkb<N>. Report, open, close, finish, reset and zone append are
mapped to the matching NVMe commands. It needs:
a backend that passes NVMe commands through:
io_uring_cmd,spdk,libvfnorupcie, oremu/thrpoolover thenvme,spdk,libvfnorupciesync path. Others are refused.a kernel with
CONFIG_BLK_DEV_ZONEDandUBLK_F_ZONED(6.6+), and matching<linux/ublk_cmd.h>at build time; the kernel ties it to user copy.a power-of-two zone size and a capacity of whole zones.
The open and active limits are MOR + 1 and MAR + 1, and the append limit
is ZASL capped by the per-IO buffer and the zone size. The kernel may
publish other values in /sys/block/ublkb<N>/queue. Discard and write zeroes
are not advertised: a deallocate is not reconciled with the write pointers.
A zone report must fit one request, or ublk_drv (6.6 to 6.18) returns it
scrambled. The per-IO buffer holds 64 bytes per zone, so --max-io-bytes
should be at least 64 times the zone count, rounded up to a page; qublk
warns at start-up when it is not, and when MDTS rules it out.
Zone management or append through the /dev/ublkb<N>-nvme character device
bypasses the block layer’s write pointer tracking, and the next write to that
zone may fail with EIO. Use it for read-only commands only.
Example — ZNS namespace via the NVMe character device:
xnvme qublk serve /dev/ng1n1 --be io_uring_cmd --qdepth 64 --nqueues 4
blkzone report /dev/ublkb0
mkfs.btrfs -O zoned /dev/ublkb0
del — Delete a leftover device#
An xnvme qublk serve that is killed rather than signalled cleanly leaves its ublk
device behind. del stops and deletes such a device by identifier:
xnvme qublk del --dev-id 0
Usage: xnvme qublk del [<args>]
Delete a ublk device left behind by a killed server
Where <args> include:
[ --dev-id NUM ] ; Use given 'NUM' as device identifier
[ --help ] ; Show usage / help
See 'xnvme qublk --help' for other commands
qublk - ublk server backed by xNVMe -- ver: {major: 0, minor: 7, patch: 5, vcs: 'e6cf2f4'}