CI#
An overview of the environments and the virtual, and physical resources utilized for the xNVMe CI is illustrated below.
xNVMe CI environments and resources#
The above figure is outdated, as bench and verify have been migrated to a MaaS provider and are no longer on-premises. However, one system remains on-prem because no MaaS provider currently offers a suitable replacement.
Infrastructure#
The main logical infrastructure component for the xNVMe CI is GitHub Actions (GHA). GHA handles events occuring on the following repositories:
And decides what to execute and where. In other words GHA is utilized as a resource-scheduler and pipeline-engine. The executor role is delegated to CIJOE for details, then have a look at CIJOE in xNVMe.
The motivation for this separation is to make it simpler to reproduce build, test, and verfication issues occuring during a CI run, using locally available resources, by executing the CIJOE in xNVMe tasks and scripts.
Jobs#
The jobs performed by the xNVMe CI catch the following issues during integration of changes / contributions:
Code format issues
Linting and code-formating
clang-format for C
clippy for Rust
black / ruff for Python
Build issues
Build with debug enabled and disabled
Detect linking issues with both the static and shared library
Tested on every OS listed in the Toolchain section
Functional regressions
Running logical tests exercising all code-paths
Using a naive “ramdisk” backend
Using emulated NVMe devices via qemu
Using physical machines
In addition to cathing issues, then the CI is also utilized for:
Benchmarking of xNVMe
Using physical machines
Measure peak IOPS for a single physical CPU core
Specifically for the integration of xNVMe in SPDK (
bdev_xnvme)
Statically Analyze the C code-base
CodeQL via GitHub
Coverity
Produce and deploy documentation
Run all example commands (
.cmdfiles) and collect their output in.outfilesRender the Sphinx-doc documentation as HTML
Upload rendered documentation to xnvme.io via GitHub-pages
These following sections provide system-setup notes and other details for the various CI jobs.
When the jobs run, and what is published#
The verify workflow runs on pull-requests, and otherwise weekly for the
benchmarks, when started by hand, and for the documentation, which
publish-docs and publish-release start. What a pull-request needs to
merge runs there once: a fixed set of jobs on every run,
and behind a gate: label, read when the run starts, the jobs in QEMU
guests, on lab machines, the benchmarks and the documentation with its
commands run (see What CI runs: always, and behind a gate). The check ci-result
says whether what ran passed. Main and its tags run none of it again but the
documentation. A newer run of a pull-request cancels the one in
progress, as does a newer run of a branch started the same way, by the
schedule or by hand for the same job.
Each kind of job has a workflow file of its own, build.yml,
verify-emulated.yml for NVMe/PCIe and NVMe/TCP,
verify-nvme-rdma-emulated.yml, verify-gpu-emulated.yml,
verify-nvme-pcie-hardware.yml, verify-makefile.yml,
verify-packaging.yml, perf.yml and docgen-full.yml.
verify calls them, so they run as part of its run and use its artifacts,
and its ci-result checks every one that ran.
A push to main runs two workflows instead. publish-docs starts a
docgen run of verify on main, which builds the documentation of main’s
tree and deploys it to xnvme.io via GitHub-pages. When the head commit is
ver: bump to vX.Y.Z, publish-release finds the pull-request that was
merged and the verify run of its head that ci-result passed, tags main
vX.Y.Z and creates the release, with the source archives, the Python sdist and the test
results of that run, and the CHANGELOG.rst section of the version as its
notes, and starts a docgen run on the tag for the documentation of the
release. The artifacts are used only when the pull-request’s head has the tree
main now has, which branch protection makes the case by requiring it to be up
to date with main; otherwise the release fails, see
Release Checklist.
Any job can be run by hand: start the verify workflow from the Actions tab,
on a branch, with job set to all for every job, or to verify,
perf or docgen. A manual docgen run deploys the
documentation of its branch.
Where the devices come from, and which guest runs what#
A verification job gets its NVMe devices from one of three sources:
emulated: QEMU and the vfio-user device servers in xnvme-devenv’s host container, on a GitHub-hosted runner, with the guest booted on a rig
passthru: a lab machine’s real drive, passed by vfio-pci to a guest booted in the same host container, on the lab machine
physical: the machine’s own OS, with no guest (the macOS machine)
The guest follows what is verified. Debian 13 is kept upstream-pure, a stock kernel and in-tree drivers only, and verifies the transports on a kernel users run. The xnvme-devenv Ubuntu 26.10 guest carries a patched kernel and the drivers built for it, matched to the device servers in the host container of the same release, and verifies what needs them; a driver and a server from different releases present as a wedged device rather than an error.
Verifies |
Emulated |
Passthru |
Physical |
|---|---|---|---|
NVMe/PCIe |
|
|
|
NVMe/TCP |
|
none |
none |
NVMe/RDMA |
|
none |
none |
NVMe with buffers in GPU memory |
|
none |
none |
A cell that says none has no job: that subject is not verified from that source.
The benchmarks run on the passthru machines, weekly, on request, and on a
pull-request labelled gate:perf, where ci-result counts them.
Containers and guest images#
The emulated CI jobs and docgen-full run inside a Docker host container
(set by GHA’s container: directive), and boot a QEMU guest as part
of their work. The host container comes from xnvme-devenv, which adds the
vfio-user device servers to a nosi base; the guest images come from
nosi itself.
Both publish OCI artifacts on GHCR.
Host containers carry the toolchain the GHA job itself uses: QEMU binaries, CIJOE, devbind / hugepages helpers, and
vfu_kvssd. They also carry the rigs a guest boots with, its PCIe topology and NVMe controllers, and the scripts that boot them. The QEMU jobs bootstorage-node-emulated;make guest-start RIG=<rig> ...takes others, rig names or files, each layered over the one before.Guest disk images are the qcow2 boot disk the in-job QEMU instance runs. They are pulled in-job by the
stage_nosi_guestCIJOE script (invoked fromguest.yamlright beforeguest_initialize), which talks directly to the OCI Distribution API via stdliburllib(noorasCLI dependency); the blob digest is pinned asoras.urlunder[system-imaging.images.<name>]incijoe/configs/system_imaging.toml. The nosi guests ship with root locked and an operator accountodus;cijoe/scripts/root_unlock.pyruns right afterguest_startto open root SSH so the rest of the CIJOE task continues as root unchanged (Linux useschpasswd+systemctl, FreeBSD usespw usermod+service sshd reload).Distro-deps containers are an image per Linux distribution with xNVMe’s dependencies, as its
toolbox/pkgsscript installs them, for the build jobs and the source and binding jobs to run in, and for anyone to try a distribution in. Thedistro-depsworkflow builds them fromtoolbox/pkgs/Dockerfileasghcr.io/xnvme/xnvme/distro-deps-<os>-<ver>:main: on a pull-request changing the scripts without pushing, and onmain, when a script changes and every 12 hours, pushing them.
Job |
Host container |
Guest disk image |
|---|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Windows 11, from the Hetzner Storage Box, on the runner |
|
|
|
The emulated jobs and docgen-full run in the xnvme-devenv host container, with
upstream QEMU; the NVMe KV command set is supplied by vfio-user-kvssd
over vfio-user, installed in the container. verify-nvme-pcie-hardware / passthru and the
benchmarks run the container on the lab machine, and pass its drive through
with PASSTHRU=0000:02:00.0; they mount the machine’s
/home/ghr/system_imaging, so a guest image is staged once per machine, and
the Windows image, which cannot be distributed, is there.
Reproducing a job, and developing in it#
The emulated jobs run the same make targets you run locally, in the same host container; Developing in a guest describes them.