Bench#

xNVMe CI Resource Overview

xNVMe CI environments and resources#

This describes the setup / notes

System Notes#

Since the system utilized is a custom physical setup, then notes are provided here describing the hardware utilized and the configuration performed to get the hardware resources into their state of function.

bifrost#

The physical hardware resources utilized for benchmarking reside in an isolated network, only traffic allowed in is via a WireGuard VPN. Taking care of these firewall and VPN tasks is a Netgate 1100 running pfSense+.

The following is setup:

  • Firewall rules: default to deny

  • DHCP Server with mac-address based IP assignment

  • WireGuard Enabled

See the Netgate 1100 Manual for details on how to configures this.

bench-intel#

Hardware Specs. for bench-intel#

CPU

Memory

Motherboard

NVMe Devices

Intel 12th Gen. Core i5-12600

32 GB (2x Kingston 548C38-16)

MSI Z690-A

  • 4x 980 PRO 2TB

  • 3x 980 PRO 1TB

  • 1x 980 PRO with Heatsink 1TB

bench-intel-pikvm#

This running PiKVM on a Raspberry Pi 4b with the PiKVM V3 HAT, for setup notes see PiKVM V3 HAT Config Notes.

bench-amd#

Hardware Specs. for bench-amd#

CPU

Memory

Motherboard

NVMe Devices

AMD Ryzen Threadripper 1900X 8-Core

32 GB (4x CMK32GX4M4B3200C16)

ASRock X399 Phantom Gaming 6

  • 8x MEMPEK1W016GA

  • 2x MEMPEK1J016GAL

bench-amd-pikvm#

…

runner-rockpi-01#

This is a Rock PI 4b with a 512GB NVMe SSD attached. It is running Armbian / Bookworm.

Add a user for the github action runner, named ghr:

adduser ghr
usermod -aG sudo ghr

Install some tools:

apt-get -qy install \
        git \
        vim \
        time \
        tree

Configure the NVMe storage, by doing the following:

  • Partition and ext4-format

    • fdisk /dev/nvme0n1

    • mkfs.ext4 /dev/nvme0n1p1

  • Get the UUID

    • Run: blkid

  • Edit /etc/fstab using the UUID with mount-point at /gha

Then reload:

systemctl daemon-reload
mount /ghr
chown -R ghr:ghr /ghr

Install docker#

Do this:

curl -fsSL https://get.docker.com -o get-docker.sh
sh ./get-docker.sh

Change docker to store container-images and temporary data onto the NVMe device, to avoid wear on the emmc:

# Setup a docker config
mkdir /etc/systemd/system/docker.service.d
echo "[Service]" >> /etc/systemd/system/docker.service.d/docker.conf
echo "ExecStart=" >> "/etc/systemd/system/docker.service.d/docker.conf"
echo "ExecStart=/usr/bin/dockerd --data-root /ghr/docker" >> "/etc/systemd/system/docker.service.d/docker.conf"

# Reload it
systemctl daemon-reload
systemctl restart docker

GitHUB Runner#

Switch to the ghr user, go into the /ghr mountpoint, download and extract the github-action-runner:

su ghr
cd /ghr
mkdir actions-runner && cd actions-runner
curl -o actions-runner-linux-arm64-2.309.0.tar.gz -L https://github.com/actions/runner/releases/download/v2.309.0/actions-runner-linux-arm64-2.309.0.tar.gz
echo "b172da68eef96d552534294e4fb0a3ff524e945fc5d955666bab24eccc6ed149  actions-runner-linux-arm64-2.309.0.tar.gz" | shasum -a 256 -c
tar xzf ./actions-runner-linux-arm64-2.309.0.tar.gz

Then we create two runners, one for bench-amd, and one for bench-intel:

export RUNNERS="bench-amd bench-intel"
export RUNNER_USER=ghr
export URL=https://github.com/OpenMPDK/xNVMe
export TOKEN={SUPER_SECRET}

With the above defined, then you can execute these:

      cd /ghr

      # Setup runners
      for RUNNER_NAME in $RUNNERS; do cp -r actions-runner "runner-for-${RUNNER_NAME}"; done;

      # Register runners

      cd /ghr/runner-for-bench-intel
      ./config.sh --unattended --url ${URL} --token ${TOKEN} --labels bench,intel --replace --name runner-for-bench-intel
cd ..

      cd /ghr/runner-for-bench-amd
      ./config.sh --unattended --url ${URL} --token ${TOKEN} --labels bench,amd --replace --name runner-for-bench-amd
cd ..

Install and run them as a service:

cd /ghr

# Service(s): install
for RUNNER_NAME in $RUNNERS; do pushd "runner-for-${RUNNER_NAME}"; sudo ./svc.sh install ${RUNNER_USER}; popd; done

# Service(s): start
for RUNNER_NAME in $RUNNERS; do pushd "runner-for-${RUNNER_NAME}"; sudo ./svc.sh start; popd; done

# Service(s): status
for RUNNER_NAME in $RUNNERS; do pushd "runner-for-${RUNNER_NAME}"; sudo ./svc.sh status; popd; done

And when needing to update:

# Services: stop
for RUNNER_NAME in $RUNNERS; do pushd "runner-for-${RUNNER_NAME}"; sudo ./svc.sh stop; popd; done

# Services: uninstall
for RUNNER_NAME in $RUNNERS; do pushd "runner-for-${RUNNER_NAME}"; sudo ./svc.sh uninstall; popd; done

# Remove the runner
for RUNNER_NAME in $RUNNERS; do pushd "runner-for-${RUNNER_NAME}"; ./config.sh remove --token ${TOKEN}; popd; done;
On this page