Ktest is just a set of C headers with very close hierarchy to the original Linux, but much simplified and combined with some additional helpers. This headers provide only most usable stubs for real kernel headers to launch a relatively standalone logic under the test.

The good examples of the code suitable for the testing with the framework are data strucures like HTrie, mathematic functions like APM maths or multi-precission math.

The framework emphasises simplicity to make debugging of complicated and standalone logic as productive as possible: (re-)run quickly, easily traced with gdb and valgrind.

KUnit, available in the mainstream kernels at linux/tools/testing/kunit/, is essentially a bunch of Python scripts around User-mode Linux (UML). UML is also in the mainstream kernel and one just needs to buid the kernel for um architecture.

The most important thing bout KUnit/UML is that this is a fully functional Linux kernel, so you need a fake filesystem, login to the console and so on. It can be debugged with gdb (see example), but special steps are required. All in all, this can be and should be treated as a virtualization solution.

Tests are developed for KUnit/UML as Linux kernel modules (this LWN article provides a good intro into the framework), which is somewhat close to current tempesta_fw/t/unit framework operation. As discussed in another LWN article the framework emulates device initialization, making tests (re-)run faster than in KVM, which we use. Ext4 tests, as one of the good use case for the framework, require a filesystem anyhow and they do required significant underlaying layers (block IO) emulation. This is example of non-standalone logic, which can not be tested with our ktest framework.

Thus, KUnit/UML is closer to tempesta_fw/t/unit tests than to ktest. In future, if tempesta_fw/t/unit grows or become more complex, e.g. require the TCP stack, it could make sense to port tempesta_fw/t/unit to KUnit/UML.

Linux Kernel Library (see also LWN anritcle) build the Linux kernel as a library, so a user space application can use the Linux API. Unfortunately, the project isn’t mainstream and it looks like an overkill for us to apply and support the Tempesta patch for the LKL kernel and build the separate kernel just for the couple tests for the standalone logic.

Debugging some parts of code requires setups which contain more than one NUMA node. Here is an instruction on how to create a VM with NUMA support.

sudo apt install qemu-kvm

sudo apt install cloud-image-utils

wget https://cloud-images.ubuntu.com/releases/noble/release/ubuntu-24.04-server-cloudimg-amd64.img

qemu-img resize ubuntu-24.04-server-cloudimg-amd64.img +40G

Standard cloud image is user agnostic. To save credentials and other user information we need to create yet another disk image and connect it to our VM.

$ cat > user-data <<EOF
#cloud-config
password: <password>
chpasswd: { expire: False }
ssh_pwauth: True
EOF

cloud-localds seed.img user-data

qemu-system-x86_64 \
 -enable-kvm \
 -cpu host \
 -m 8G \
 -drive if=virtio,format=qcow2,file=ubuntu-24.04-server-cloudimg-amd64.img \
 -drive if=virtio,format=raw,file=seed.img \
 -smp 16,sockets=4,cores=4,threads=1,maxcpus=16 \
 -object memory-backend-ram,size=2G,id=m0 \
 -object memory-backend-ram,size=2G,id=m1 \
 -object memory-backend-ram,size=2G,id=m2 \
 -object memory-backend-ram,size=2G,id=m3 \
 -numa node,nodeid=0,memdev=m0,cpus=0-3 \
 -numa node,nodeid=1,memdev=m1,cpus=4-7 \
 -numa node,nodeid=2,memdev=m2,cpus=8-11 \
 -numa node,nodeid=3,memdev=m3,cpus=12-15 \
 -device virtio-net-pci,netdev=net0 \
 -netdev user,id=net0,hostfwd=tcp::2222-:22 \
 -nographic 

Login: ubuntu

Password: <password from the user-data file>

ssh -p 2222 ubuntu@localhost

On the VM run: sudo apt install numactl

numactl -H

Should see NUMA configuration: 4 nodes with 4 CPUs and 2G memory as for configuration described above.

Inside created VM install build tools, download sources and patches.

Perform standard build commands.

make -j $(nproc)

make install

Then copy bzImage and initrd.img to the host.

On the host run:

scp -P 2222 ubuntu@localhost:linux-<version>/arch/x86/boot/bzImage .

scp -P 2222 ubuntu@localhost:/boot/initrd.img-<version> .

qemu-system-x86_64 \
 -enable-kvm \
 -cpu host \
 -m 8G \
 -drive if=virtio,format=qcow2,file=ubuntu-24.04-server-cloudimg-amd64.img \
 -drive if=virtio,format=raw,file=seed.img \
 -smp 16,sockets=4,cores=4,threads=1,maxcpus=16 \
 -object memory-backend-ram,size=2G,id=m0 \
 -object memory-backend-ram,size=2G,id=m1 \
 -object memory-backend-ram,size=2G,id=m2 \
 -object memory-backend-ram,size=2G,id=m3 \
 -numa node,nodeid=0,memdev=m0,cpus=0-3 \
 -numa node,nodeid=1,memdev=m1,cpus=4-7 \
 -numa node,nodeid=2,memdev=m2,cpus=8-11 \
 -numa node,nodeid=3,memdev=m3,cpus=12-15 \
 -device virtio-net-pci,netdev=net0 \
 -netdev user,id=net0,hostfwd=tcp::2222-:22 \
 -kernel bzImage \
 -initrd initrd.img-<version>
 -nographic \
 -append "root=/dev/vda1 console=ttyS0" \
 -serial mon:stdio

At the moment we have only 3 types of tests and they are well covered by the current frameworks:

1. The most developed and the largest framework is the functional test Python framework (https://github.com/tempesta-tech/tempesta-test/). This type of the tests are most beneficial because (1) they emulate cases very close to the real life scenatios and (2) they test quite a bunch of logic at once.

2. Complex standalone logic (HTrie, TLS, math in tempesta_fw/t/unit/user_space/) are covered with the ktest.

3. Linux kernel module for unit tests (https://github.com/tempesta-tech/tempesta/tree/master/tempesta_fw/t). At the moment the module is quite small and some of the tests, e.g. HTTP parser or TfwStr, are likely to be moved to ktest framework. However, at some point we might need to develop tests which (1) require to reproduce specific and rare scenarios unreachable for the functional tests framework and (2) depend on significant Linux kernel logic unmockable by the ktest. At this moment we might need to use either LKL or KUnit/UML.