Conventions#
A brief overview of conventions used for the xNVMe C codebase.
C: format#
clang-format is utilized to format the code. There are two clang-format-option
files, one for the headers (toolbox/clang-format-h) and one for the source
toolbox/clang-format-c.
You can setup your editor to use the format-files, or run clang-format via the
toolbox: make format-all. This will format all files in the repository.
C: style#
Declare variables
At the top of code-blocks; before any logic, e.g.
at the top of a function-code-block / loop-nest / anonymous-block
for-loop “control/iterator” variables should be declared and initialized in the for-loop header.
In the code-block nearest to its utilization
in a function, the top-most scope is commonly used for argument-unpacking, and the idiomatic
int errvariable, along with auxilary variables needed by the function body in the outer-most scopevariable-declaration in nested-block-scope is encouraged; still at the top of that block-scope. For example, when a variable is only ever used in a nested-block, then declare it within that scope
Initialize variables
Using zero/constant-initialization;
struct foo = {0};To unpack arguments;
int simple = arg->is->alot->simple;Do not initialize using functions that require error-handling
Global variables
Prefix them with
g_
Compiler-specific features
For example, gnu/gcc-only is discouraged, as it hinders portability
Code should preferably translate using any of gcc/clang/icc/pgi
Exceptions to this would be code in backends utilizing features which are only available on the platform that the backend services
C: API#
The API and its backend implementations aims to be consistent with the following
Pass pointers to “objects” e.g.
xnvme_cmd_pass(*ctx, ...);Pass double-pointers for “object-initialization” e.g.
xnvme_queue(..., **queue);On success, return 0. On error, Return negative
errno.Never pass or return a structure by value; pass a pointer, and fill an output argument rather than returning one, as
xnvme_opts_set_defaults(&opts)does: a structure by value is simple in C, but not over FFIA function replaced for any of these keeps working, as a
static inlineinlibxnvme_compat.hcalling its replacement, and in a debug build warning that it is deprecated, until a later release removes it; a renamed type keeps its old name there as a#define, unless the rename turns astructinto aunion, which no alias can bridgeWho owns the memory decides the signature. Memory the caller owns is filled through an output argument, with 0 or negative
errnoreturned, asxnvme_dev_get_cmd_ctx(dev, &ctx)does. Memory the library owns and lends out is returned as a pointer, NULL witherrnoset on error, and given back with a matching call, asxnvme_queue_get_cmd_ctx()andxnvme_queue_put_cmd_ctx(), orxnvme_buf_alloc()andxnvme_buf_free()
The library is not thread-safe. Only the following control-plane calls may be made concurrently, on every backend, and only with respect to each other: a call made while another thread uses the same device, queue or buffer for I/O is not covered
xnvme_dev_open(),xnvme_dev_close(),xnvme_enumerate(),xnvme_queue_init(),xnvme_queue_term(),xnvme_buf_alloc(),xnvme_buf_realloc(),xnvme_buf_free(), theirxnvme_buf_phys_*()variants,xnvme_buf_vtophys(),xnvme_buf_register(),xnvme_buf_unregister(),xnvme_cuda_queue_create()andxnvme_cuda_queue_destroy(); one process-wide lock serialises them (lib/util/xnvme_lock.c)A queue, and the command contexts drawn from it, belongs to one thread at a time, and so do a device’s synchronous and admin command paths
The device getters that identify the device on first use, such as
xnvme_dev_get_geo()andxnvme_dev_get_ident(), must not make that first call from several threads at onceBackend code that reaches the same process-wide state from threads of its own takes the same lock,
xnvme_lock()
Be minimal with definitions in the public API
E.g. the ‘sys/queue.h’ was removed from the public API as it gave several head-aches when building xNVMe on multiple platforms. However, internally, specifically in backend implementations, assumptions can be made on the availability of certain headers and their general availability
Each function in the API must have a command-line tool exercising it
The source-code for the command-line tools serve as examples of utilizing the C API in general and the tools are used during testing
The command-line utilities must use the
libxnvme_cli.has this provides a common command-line interface, the common-cli is nice as for usability, such that all cli-tools use the same arguments, it is also essential for instrumentation of not just the logic of the tool but also the library backend
C: Fields of bits#
A field of bits in a structure of the specification, such as flbas of
the Identify Namespace data, is an integer of the width the specification
gives it, not a C bit-field, whose layout is the compiler’s choice. The
integer is named as the specification names the field, such as status
of the Completion Queue Entry, not by a placeholder such as val.
Where the specification names only the fields within an integer, each
narrower than a byte and reached through a mask, the integer is named
bits, such as bits of the Extended LBA Format. A structure holding
more than one such integer names each after the fields it holds, such as
mxps_nops of the power state descriptor, and a byte with a name in common
use keeps it, such as flags of a command, as the Linux NVMe driver names
it. The integer is not wrapped in a structure of its own, unless it is the
element of an array, which needs a type for its masks to belong to, as
struct xnvme_spec_elbaf is.
Each of its fields is named by a mask, a
#defineof the bits in their place, read and written withxnvme_bits_get()andxnvme_bits_set()The mask is named
XNVME_SPEC_<STRUCT>_<FIELD>_<NAME>: the structure without itsxnvme_spec_prefix, the integer, and the name the specification gives the bits, such asXNVME_SPEC_IDFY_NS_FLBAS_FORMATfor bits 3:0 offlbasThe masks follow the structure they belong to, in the order of its integers, and of the bits within each
Each mask has a doc comment saying what the bits are, then where they are:
/** <what>; bits <hi>:<lo> of '<field>' */, orbit <n>for one bitThe integer’s own doc comment names them:
///< <what>, see XNVME_SPEC_<STRUCT>_<FIELD>_*tests/spec.cchecks, for each integer, that the structure still has it and that its masks fit its width, so a field renamed or narrowed without its masks fails the build; a new field of bits gets a line there
C: Code organisation#
Where code lives and what it is named follow from what it does, so new code has one place and one name. These rules are for people and coding agents alike: follow them rather than the nearest existing example, and where a case is not covered, raise it in review instead of inventing a convention.
Layout#
include/holds the public headers,libxnvme*.h, and nothing elselib/<group>/holds the implementation of the public API, wherelib/<group>/xnvme_<topic>.cimplementsinclude/libxnvme_<topic>.h, next to the private headers it shares,xnvme_<topic>.h. The groups are the ones the API documentation uses,NAMESPACESindocs/tooling/xnvme_docs/apigen.py, so a new public header goes in bothutil: utilities the other groups use: the lock, CRCs, LBA helpers, the version and the build configurationcore: devices, buffers, commands, queues, geometry, identifiers and optionsnvme: the NVMe command sets, the spec and its printers, topology and protection informationfile: the file APIcli: the command-line library the tools share
lib/meson.buildlistsutilfirst, thenbackendandplatform, which the public API is built on, thencore,nvme,fileandcliA printer, a function named by what it prints and suffixed
_fpr,_pror_str, is declared ininclude/libxnvme_pp.h, not in the header of its type, so that a binding leaves out the printers by leaving out one header. The printers of the command-line library stay inlibxnvme_cli.h, and astatic inlineprinter, which has no symbol for a binding to call, stays with what it prints, such asxnvme_timer_pr()lib/cuse/is a static library of its own for the tools, not part oflibxnvme. It holds the NVMe-driver ioctl() mimic; the CUSE protocol under it comes from uFuse,subprojects/ufuse.wraplib/platform/holds what selects among the backend configs:xnvme_platform.c, the device-open loop, and per OSlib/platform/<os>/xnvme_platform_<os>.c, the scan, the classify and the list of configs the OS offerslib/backend/holds the backend interface,xnvme_be.h, its shared code,xnvme_be.c,xnvme_be_cref.candxnvme_be_nosys.c, the declarations of every config,xnvme_be_registry.h, the state union,xnvme_be_state.h, and one directory per mechanism or OS
The backend interface#
A backend config, struct xnvme_be_config, combines one implementation of
each backend interface:
dev : device handles and device enumeration
mem : memory allocation for command-payloads aka buffers
admin : synchronous admin-command processing
sync : synchronous I/O command processing
async : asynchronous I/O command submission/completion via queues
Directories in lib/backend/#
A directory is either one mechanism, named after the library or interface it uses, such as
liburing,libaio,spdk,upcie,iocp,posixorthrpool, or one OS’s own interfaces,linux,freebsd,macosandwindowsA mechanism gets its own directory even when only one OS has it, as
liburingandiocpdoWhat an OS provides itself, such as its NVMe-driver ioctls, stays in that OS’s directory even when every OS has an equivalent, since the equivalents share no code
Names#
The prefix names what a file or symbol implements, and the rest names which
one. Basenames are unique across lib/, so every private header shares one
include path.
g_xnvme_be_<dir>_<interface>[_<ident>]is an implementation of a backend interface, inlib/backend/<dir>/xnvme_be_<dir>_<interface>[_<ident>].c, one implementation per file. The<ident>comes after the interface and says what sets an implementation apart from the directory’s own. The directory’s own implementation of an interface carries none, such asg_xnvme_be_liburing_async, and neither does an only implementation that is plainly the directory’s, as FreeBSD’sadminandsyncare. Another implementation of the same interface carries one, such asg_xnvme_be_liburing_async_ucmdbeside it, and so does an only implementation that is not the plain one, such asmem_hugepageordev_dmabufExample:
g_xnvme_be_liburing_async_ucmd, inlib/backend/liburing/xnvme_be_liburing_async_ucmd.cExample:
g_xnvme_be_upcie_dev_cuda, inlib/backend/upcie/xnvme_be_upcie_dev_cuda.c
xnvme_be_<dir>_<topic>.cholds, and defines nostruct xnvme_be_<interface>:code that more than one file of the directory uses, such as
xnvme_be_linux_nvme.cfor the NVMe-driver ioctl wrappercode a file outside the directory uses, so the headers it needs stay in the directory, such as
xnvme_be_linux_uapi.cforxnvme_libconf.ca part of one implementation of a hundred lines or more, such as
xnvme_be_upcie_cplane_serve_conn.candxnvme_be_upcie_cplane_share.cof the cplane serverthe functions of the public API a backend implements, such as
xnvme_cuda_queue_create()inxnvme_be_upcie_cuda_gpu.c
Code that only one file uses, and that is shorter than that, is
staticin that filexnvme_be_<dir>.his the directory’s header: theexterndeclarations of its implementations and of its shared functions. A header that configs outside the directory include, such asxnvme_be_liburing.h, keeps what needs the directory’s external library behind itsXNVME_BE_<DIR>_ENABLED, so it builds where the library is absentA topic whose declarations need headers the rest of the directory must not see, such as a GPU runtime, gets
xnvme_be_<dir>_<topic>.h
xnvme_be_<dir>_state.hholds the directory’s state,struct xnvme_be_<dir>_state. It needs no headers beyond the C library andxnvme_be.h: a type from a library or SDK is held behind a pointer or an integer of its size, solib/backend/xnvme_be_state.hcan include every one of themA function is
staticunless another file needs it, and is then namedxnvme_be_<dir>_<function>()and declared inxnvme_be_<dir>.h, such asxnvme_be_posix_sync_cmd_io(). The exception is a function of the public API that a backend implements, such asxnvme_cuda_queue_create()A config is named after the name
--beselects it by, with-as_. One whose parts all come from one directory lives inlib/backend/<dir>/xnvme_be_<dir>_configs.casg_xnvme_be_<name>, such asg_xnvme_be_spdkandg_xnvme_be_upcie_cuda. One that combines directories lives with the platform that offers it, inlib/platform/<os>/xnvme_platform_<os>_configs.c, asg_xnvme_platform_<os>_<name>, such asg_xnvme_platform_linux_libaio_bdev. Every config is declared inlib/backend/xnvme_be_registry.hA platform’s files are
xnvme_platform_<os>.candxnvme_platform_<os>_configs.cA mechanism directory has one build option,
with-<dir>, and one macro,XNVME_BE_<DIR>_ENABLED, whichxnvme env librarylists. An OS directory is built with its platform,XNVME_PLATFORM_<OS>_ENABLED. A feature within a directory isXNVME_BE_<DIR>_<FEATURE>_ENABLED, such asXNVME_BE_LINUX_BLOCK_ENABLED; one outside the backend isXNVME_<FEATURE>_ENABLED, such asXNVME_LIBISAL_ENABLED
The backend state#
All implementations in a config share one state,
dev->be.state, and the config’s dev implementation decides its layout.lib/backend/xnvme_be_state.hlists every layout as a member ofunion xnvme_be_state, and implementations reach it throughxnvme_be_state(dev), as in&xnvme_be_state(dev)->linux_state. Nothing castsdev->be.stateWhere implementations from another directory read the state, the layouts share a first member, so the read is defined by C’s rule on common initial sequences, and the state header asserts it
The posix and shim implementations read
posix_state, so the Linux, FreeBSD and macOS states embedstruct xnvme_be_posix_stateas their first member,posix, asserted withXNVME_BE_POSIX_STATE_FIRST()xnvme_platform_dev_open()stores a shared controller inctrlr, so the states of the dev implementations that share controllers start withctrlr, asserted withXNVME_BE_CTRLR_STATE_FIRST()
Dependencies#
A directory bound to an OS, such as liburing, libaio, libvfn or iocp, is built only there, and its option is off elsewhere rather than an error, since a distribution build turns every auto feature on. posix cannot be disabled on Linux, FreeBSD and macOS, whose directories take their memory and psync from it, and the configure step fails when it is
A disabled directory keeps its implementations, compiled to the functions of
xnvme_be_nosys.h. A config using a disabledemu,nil,thrpoolorshimstays and reports-ENOSYS; one using any other disabled directory is left out of the platform’s list.nosysis for a directory that is switched off, never a way to leave a broken dependency to fail at runtimeA directory includes another directory’s header only for what it uses
Where new code goes#
Part of the public API:
lib/<group>/xnvme_<topic>.c, declared ininclude/libxnvme_<topic>.h, withxnvme_<topic>added to the same group inNAMESPACESA new library or mechanism: a new directory,
lib/backend/<name>/, with its header, its state header, one file per implementation, ameson.buildadding its sources and include directory, its optionwith-<name>and macroXNVME_BE_<NAME>_ENABLED, its entry ing_libconf_partsinlib/util/xnvme_libconf.c, whichxnvme env libraryprints, and its configs if they need nothing from other directoriesSomething an OS provides itself: that OS’s directory,
lib/backend/<os>/A config combining directories: the platform’s
xnvme_platform_<os>_configs.c, listed in.backendsof the platform’sstruct xnvme_platforminxnvme_platform_<os>.cand declared inxnvme_be_registry.h