Getting Started#
Here you will be taken through the process of Building and installing xNVMe, which includes retrieving the xNVMe source, installing a C compiler / toolchain, libraries, and then building and installing xNVMe itself. With xNVMe in place on your system, an Example Program is provided for you to Compile, link, and run!.
Building and installing#
Below are examples of building and installing xNVMe on different operating systems. Additional examples and detailed information can be found in the Toolchain section, covering numerous Linux distributions, as well as FreeBSD, macOS, and Windows versions.
# Clone the xNVMe repos
git clone https://github.com/xnvme/xnvme.git
cd xnvme
# Install toolchain and libraries on Linux (Debian Bookworm)
sudo ./toolbox/pkgs/debian-bookworm.sh
# configure xNVMe and build meson subprojects(SPDK)
meson setup builddir
# build xNVMe
meson compile -C builddir
# install xNVMe
sudo meson install -C builddir
# uninstall xNVMe
# cd builddir && meson --internal uninstall
# Clone the xNVMe repos
git clone https://github.com/xnvme/xnvme.git
cd xnvme
# Install toolchain and libraries on FreeBSD (13)
sudo ./xnvme/toolbox/pkgs/freebsd-13.sh
# configure xNVMe and build meson subprojects(SPDK)
meson setup builddir
# build xNVMe
meson compile -C builddir
# install xNVMe
sudo meson install -C builddir
# uninstall xNVMe
# cd builddir && meson --internal uninstall
# Clone the xNVMe repos
git clone https://github.com/xnvme/xnvme.git
cd xnvme
# Install toolchain and libraries on macOS (14)
sudo ./xnvme/toolbox/pkgs/macos-14.sh
# configure xNVMe
meson setup builddir
# build xNVMe
meson compile -C builddir
# install xNVMe
sudo meson install -C builddir
# uninstall xNVMe
# cd builddir && meson --internal uninstall
# Clone the xNVMe repos
git clone https://github.com/xnvme/xnvme.git
cd xnvme
# Install toolchain and libraries on Windows (2022)
Set-ExecutionPolicy Bypass -Scope Process -Force
.\toolbox\pkgs\windows-2022.ps1
# Setup helper-function to invoke commands inside msys2 environment from Powershell
function run { param([string]$cmd=$(throw "You must specify a command.")); & msys2_shell -no-start -here -use-full-path -defterm -mingw64 -c "$cmd" }
# configure xNVMe (adjust this to correct path)
run "meson setup builddir --prefix=/usr"
# build xNVMe
run "meson compile -C builddir"
# install xNVMe
run "meson install -C builddir"
# uninstall xNVMe
# cd builddir && run "meson --internal uninstall"
Note
Run this in an elevated Powershell session!
Building system software can be challenging. If you encounter errors when following the steps above, and if meson is new to you, a couple of pointers are provided to help you out. You can also look at the Troubleshooting section for known issues.
- Build-errors
Details on the build errors can be seen by inspecting the log file at
builddir/meson-logs/meson-log.txt
.- Rebuilding
In case rebuilding fails, e.g., due to errors during
meson setup
or missing toolchain/dependencies, remove thebuilddir
folder. If errors persist, try also removing the SPDK subproject atsubprojects/spdk
.- Customizing-the-build
If you want to customize the build, e.g., install into a different location, this is handled by meson built-in options. In addition, you can inspect
meson_options.txt
which contains options specific to xNVMe.
Example Program#
This example code illustrates how to use the xNVMe library (libxnvme
)
in C to perform the following tasks with an NVMe device: open the device,
probe and retrieve device information, print device information, and finally
close the device.
#include <errno.h>
#include <libxnvme.h>
int
main(int argc, char **argv)
{
struct xnvme_opts opts = xnvme_opts_default();
struct xnvme_dev *dev;
if (argc < 2) {
xnvme_cli_perr("Usage: %s <identifer>", EINVAL);
return 1;
}
dev = xnvme_dev_open(argv[1], &opts);
if (!dev) {
xnvme_cli_perr("xnvme_dev_open()", errno);
return 1;
}
if (xnvme_dev_derive_geo(dev)) {
printf("Failed deriving geometry, check your permissions.");
}
xnvme_dev_pr(dev, XNVME_PR_DEF);
xnvme_dev_close(dev);
return 0;
}
Additional examples of using the xNVMe C API can be found in the xNVMe repository. This includes explicit example code, the source for the command-line tools, and tests. For examples of efficient and high-performance integration, refer to the fio xNVMe I/O engine and the SPDK xNVMe bdev module.
Furthermore, while the example given above is written in C, bindings for the xNVMe C API are also available for other languages. These include Python ( Pypi ) bindings and Rust ( crates.io ) bindings.
Compile, link, and run!#
xNVMe provides a pkg-config file (xnvme.pc
),
it is populated to fit the version of xNVMe installed on the system. Use it
to ensure you get the correct flags. For example, like so:
# Build, link and make the example executable
gcc ../getting_started/hello.c $(pkg-config --libs xnvme) -o hello
chmod +x hello
# Run it
./hello /dev/nvme2n1
When doing so, it will print output equivalent to the following.
# Output from "./hello /dev/nvme2n1"
xnvme_dev:
xnvme_ident:
uri: '/dev/nvme2n1'
dtype: 0x2
nsid: 0x1
csi: 0x0
subnqn: 'nqn.1994-11.com.samsung:nvme:PM1733:2.5-inch:S537NE0MB00092 '
xnvme_be:
admin: {id: 'nvme'}
sync: {id: 'nvme'}
async: {id: 'emu'}
attr: {name: 'linux'}
xnvme_opts:
be: 'linux'
mem: 'posix'
dev: 'FIX-ID-VS-MIXIN-NAME'
admin: 'nvme'
sync: 'nvme'
async: 'emu'
xnvme_geo:
type: XNVME_GEO_CONVENTIONAL
npugrp: 1
npunit: 1
nzone: 1
nsect: 7501476528
nbytes: 512
nbytes_oob: 0
tbytes: 3840755982336
mdts_nbytes: 65024
lba_nbytes: 512
lba_extended: 0
ssw: 9
pi_type: 0
pi_loc: 0
pi_format: 0
# Build, link and make the example executable
gcc ../getting_started/hello.c $(pkg-config --libs xnvme) -o hello
chmod +x hello
# Detach NVMe-devices from OS NVMe driver and bind to vfio-pci / uio-generic
sudo xnvme-driver
# Run it
sudo ./hello 0000:05:00.0
When doing so, it will print output equivalent to the following.
# Output from "xnvme-driver"
0000:01:00.0 (144d a80c): nvme -> vfio-pci
0000:02:00.0 (144d a80c): Active mountpoints on /dev/nvme0n1, so not binding PCI dev
0000:05:00.0 (144d a824): nvme -> vfio-pci
# Output from "sudo ./hello 0000:05:00.0"
xnvme_dev:
xnvme_ident:
uri: '0000:05:00.0'
dtype: 0x2
nsid: 0x1
csi: 0x0
subnqn: 'nqn.1994-11.com.samsung:nvme:PM1733:2.5-inch:S537NE0MB00092 '
xnvme_be:
admin: {id: 'nvme'}
sync: {id: 'nvme'}
async: {id: 'nvme'}
attr: {name: 'spdk'}
xnvme_opts:
be: 'spdk'
mem: 'spdk'
dev: 'FIX-ID-VS-MIXIN-NAME'
admin: 'nvme'
sync: 'nvme'
async: 'nvme'
xnvme_geo:
type: XNVME_GEO_CONVENTIONAL
npugrp: 1
npunit: 1
nzone: 1
nsect: 7501476528
nbytes: 512
nbytes_oob: 0
tbytes: 3840755982336
mdts_nbytes: 1048576
lba_nbytes: 512
lba_extended: 0
ssw: 9
pi_type: 0
pi_loc: 0
pi_format: 0
# Build, link and make the example executable
gcc ../getting_started/hello.c $(pkg-config --libs xnvme) -o hello
chmod +x hello
# Run it
./hello /dev/nvme0ns1
When doing so, it will print output equivalent to the following.
# Output from "./hello /dev/nvme0ns1"
xnvme_dev:
xnvme_ident:
uri: '/dev/nvme0ns1'
dtype: 0x2
nsid: 0x1
csi: 0x0
subnqn: 'nqn.1994-11.com.samsung:nvme:PM1733:2.5-inch:S537NE0MB00092 '
xnvme_be:
admin: {id: 'nvme'}
sync: {id: 'nvme'}
async: {id: 'emu'}
attr: {name: 'fbsd'}
xnvme_opts:
be: 'fbsd'
mem: 'posix'
dev: 'FIX-ID-VS-MIXIN-NAME'
admin: 'nvme'
sync: 'nvme'
async: 'emu'
xnvme_geo:
type: XNVME_GEO_CONVENTIONAL
npugrp: 1
npunit: 1
nzone: 1
nsect: 7501476528
nbytes: 512
nbytes_oob: 0
tbytes: 3840755982336
mdts_nbytes: 1048576
lba_nbytes: 512
lba_extended: 0
ssw: 9
pi_type: 0
pi_loc: 0
pi_format: 0
# Build, link and make the example executable
gcc ../getting_started/hello.c $(pkg-config --libs xnvme) -o hello
chmod +x hello
# Run it
./hello /dev/disk4
When doing so, it will print output equivalent to the following.
# Output from "./hello /dev/disk4"
xnvme_dev:
xnvme_ident:
uri: '/dev/disk4'
dtype: 0x3
nsid: 0x1
csi: 0x0
subnqn: 'nqn.1994-11.com.samsung:nvme:PM1733:2.5-inch:S537NE0MB00092 '
xnvme_be:
admin: {id: 'nvme'}
sync: {id: 'psync'}
async: {id: 'emu'}
attr: {name: 'macos'}
xnvme_opts:
be: 'macos'
mem: 'posix'
dev: 'FIX-ID-VS-MIXIN-NAME'
admin: 'nvme'
sync: 'psync'
async: 'emu'
xnvme_geo:
type: XNVME_GEO_CONVENTIONAL
npugrp: 1
npunit: 1
nzone: 1
nsect: 7501476528
nbytes: 512
nbytes_oob: 0
tbytes: 3840755982336
mdts_nbytes: 1048576
lba_nbytes: 512
lba_extended: 0
ssw: 9
pi_type: 0
pi_loc: 0
pi_format: 0
# Build, link and make the example executable
sh "gcc ..\getting_started\hello.c -I..\..\toolbox\third-party\windows $(pkg-config --libs xnvme) -o hello"
# Run it
.\hello.exe .\PhysicalDrive1
When doing so, it will print output equivalent to the following.
# Output from ".\hello.exe \\.\PhysicalDrive1"
xnvme_dev:
xnvme_ident:
uri: '\\.\PhysicalDrive1'
dtype: 0x2
nsid: 0x1
csi: 0x0
subnqn: 'nqn.1994-11.com.samsung:nvme:PM1733:2.5-inch:S537NE0MB00092 '
xnvme_be:
admin: {id: 'nvme'}
sync: {id: 'nvme'}
async: {id: 'iocp'}
attr: {name: 'windows'}
xnvme_opts:
be: 'windows'
mem: 'windows'
dev: 'FIX-ID-VS-MIXIN-NAME'
admin: 'nvme'
sync: 'nvme'
async: 'iocp'
xnvme_geo:
type: XNVME_GEO_CONVENTIONAL
npugrp: 1
npunit: 1
nzone: 1
nsect: 7501476528
nbytes: 512
nbytes_oob: 0
tbytes: 3840755982336
mdts_nbytes: 1048576
lba_nbytes: 512
lba_extended: 0
ssw: 9
pi_type: 0
pi_loc: 0
pi_format: 0
Note
Run this in an elevated Powershell session!
As the compilation and execution of the Example Program demonstrate, platform and interface-specific concerns exist. However, the xNVMe API and library encapsulate these differences, enabling the same example code to function uniformly across all platforms. This provides a unified platform and system interface abstraction.
The library encapsulation in xNVMe is referred to as Backends. In the example above, only a fraction of the supported system interfaces are shown. The section on Backends provide additional documentation on this, as well as describing other runtime concerns such as system setup and configuration.
With the library and toolchain in place, you might want to explore the Tools and API. For this, you can use your own storage devices. However, you can also look at the Tutorials section, specifically the Linux Dev. Environment section, which will enable you to develop and test using emulated storage devices with support for features such as such as Zoned Namespaces, Key-Value, and Flexible-Data-Placement.
In case things aren’t working for you then have a look at the following section on Troubleshooting.
Troubleshooting#
This section is here to give you pointers to troubleshoot whether xNVMe or your system is misbehaving. First off ensure that you are you have the setup your system, please refer to the Toolchain section for details on installing compilers, libraries etc. and the Backends section on notes on system configuration and runtime instrumentation of the library backends.
Should you not find the answers you are looking here, then please reach out by raising an issue or go to Discord for synchronous interaction.
Build Errors#
xNVMe adheres to best practices and conventions for system software. It links with other libraries available on the system and downgrades functionality to disable dependent modules if a library is unavailable. However, xNVMe diverges in one area: the subproject build of SPDK. In this case, xNVMe retrieves the SPDK repository, patches the source, and builds it for linkage with xNVMe. Consequently, most reported issues are related to this process and are addressed accordingly.
Shallow source-archive#
If you are getting errors while attempting to configure and build xNVMe then it is likely due to one of the following:
You are building in an offline environment and only have a shallow source-archive or a git-repository without subprojects.
The full source-archive is made available with each release and downloadable from the GitHUB Release page release page. It contains the xNVMe source code along with all the third-party subproject source of SPDK.
Missing dependencies / toolchain#
You are missing dependencies, see the Toolchain for installing these on FreeBSD and a handful of different Linux Distributions
Once you have the full source of xNVMe, third-party library dependencies, and setup the toolchain then run the following to ensure that the xNVMe repository is clean from any artifacts left behind by previous build failures:
make clobber
And then go back to the Building and installing and follow the steps there.
Note
When running make clobber
then everything not comitted is “lost”. Thus,
if you are developing/modifying xNVMe, then make you commit of stash your
changes before running it.