KCIDB-ng is fresh rewrite of our wrappers around the KernelCI Database (KCIDB). It provides services for submitting kernel test data to the KCIDB database via HTTP requests, and also come with log analysis capabilities.

The system consists of several interconnected components:

1. kcidb-restd-rs - A Rust-based REST service that:

   • Receives JSON submissions via HTTP/HTTPS
   • Authenticates users via JWT
   • Stores valid submissions in a spool directory
   • Provides status endpoints for submissions

2. ingester - A Python service that:

   • Processes JSON submissions from the spool directory
   • Validates them against the KCIDB schema
   • Loads them into the KCIDB database
   • Archives processed submissions This service has been replaced with KernelCI Dashboard's implementation of the ingester and was left here just as a temporary and standalone usage.

3. logspec-worker - A Python service that:

   • Monitors the database for failed tests and builds
   • Downloads and analyzes log files using the logspec library
   • Identifies issues and incidents from logs
   • Submits findings back to KCIDB

4. PostgreSQL Database - Stores all KCIDB data

   • Can be run locally (self-hosted mode)

• Docker and Docker Compose (the newer docker compose plugin, not the legacy docker-compose; version 2.0+ required)
• Git

git clone https://github.com/kernelci/kcidb-ng.git
cd kcidb-ng

The legacy ingester/ingester.py implementation is deprecated and kept for compatibility only. Current compose setups use the dashboard-backed ingester (monitor_submissions), built from Dockerfile.django-ingester. Planned removal of the deprecated ingester is April 30, 2026.

To quickly start the KCIDB-ng services with a local PostgreSQL database, run:

./self-hosted.sh run

This script will:

• Build and start the Docker containers
• Initialize the PostgreSQL database
• Start the REST API, dashboard ingester, and logspec-worker services

To run from local Dockerfiles (no GHCR image pulls), use:

./self-hosted.sh --dev run

Also available commands:

• ./self-hosted.sh down - Stops the services
• ./self-hosted.sh clean - Stops and removes all containers, configs, databases, networks, and volumes

Create a .env file in the root directory with the following environment variables:

# PostgreSQL configuration
POSTGRES_PASSWORD=kcidb
PS_PASS=kcidb
PG_URI=postgresql:dbname=kcidb user=kcidb_editor password=kcidb host=db port=5432
# JWT authentication
JWT_SECRET=your_jwt_secret

The self-hosted profile includes a local PostgreSQL database and an initialization service:

docker compose --profile=self-hosted up -d --build

This command:

• Builds and/or pulls required images (as defined by compose files)
• Sets up a local PostgreSQL database
• Initializes the database schema
• Starts the REST API, dashboard ingester, and logspec-worker services

Note: By default it is expecting PostgreSQL to be running with default settings, except postgres password which is set to kcidb. It will also create a user kcidb_editor with password kcidb and a database kcidb, and user kcidb_viewer with password kcidb for read-only access.

This project now has two compose workflows:

• Default mode uses images from GitHub Container Registry for kcidb-rest and ingester (as defined in docker-compose.yaml / docker-compose-all.yaml).
• Local development mode rebuilds those images from local Dockerfiles by adding docker-compose-dev.yaml.

Run with prebuilt images:

docker compose -f docker-compose.yaml up -d
docker compose -f docker-compose-all.yaml up -d

Run local-source builds:

docker compose -f docker-compose.yaml -f docker-compose-dev.yaml up -d --build
docker compose -f docker-compose-all.yaml -f docker-compose-dev.yaml up -d --build

For self-hosted local PostgreSQL with local builds:

docker compose -f docker-compose.yaml -f docker-compose-dev.yaml --profile=self-hosted up -d --build

If your kcidb-ng is installed in isolated environment, you can disable JWT authentication by commenting out the JWT command in docker-compose.yaml:

#    command: ["/usr/local/bin/kcidb-restd-rs","-j",""]

If you want to use JWT authentication, you can generate a token using the following command:

kcidb-restd-rs/tools/jwt_rest.py --secret YOUR_SECRET --origin YOUR_ORIGIN

To validate a JWT token, you can use the following command:

curl -X GET \
  -H "Authorization: Bearer <jwt_token>" \
  https://localhost:443/authtest

This will return a JSON response with the token's validity.

To submit data to the REST API:

curl -X POST \
  -H "Authorization: Bearer <jwt_token>" \
  -H "Content-Type: application/json" \
  -d @submission.json \
  https://localhost:443/submit

You can check the status of your submission using:

curl -X GET   -H "Authorization: Bearer <yourtoken>" https://staging.db.kernelci.org/status?id=wvTu6myNQOlM7IEWhHJz8WnnE0GTG1yz
{"id":"wvTu6myNQOlM7IEWhHJz8WnnE0GTG1yz","status":"failed","message":"File found"}

Possible status values:

• {"id":"0","status":"error","message":"Empty id"} The request was invalid (e.g., missing or invalid ID, authentication error, etc.).
• {"id":"wvTu6myNQOlM7IEWhHJz8WnnE0GTG1yz","status":"inprogress","message":"File still in progress"} A submission file with .json.temp exists, indicating the upload is not yet complete.
• {"id":"wvTu6myNQOlM7IEWhHJz8WnnE0GTG1yz","status":"ready","message":"File waiting for processing"} The submission file exists and is ready for processing, but not yet archived or failed.
• {"id":"wvTu6myNQOlM7IEWhHJz8WnnE0GTG1yz","status":"processed","message":"File archived"} The submission has been processed and archived.
• {"id":"wvTu6myNQOlM7IEWhHJz8WnnE0GTG1yz","status":"failed","message":"File failed to pass validation"} The submission failed validation and is in the failed directory.
• {"id":"wvTu6myNQOlM7IEWhHJz8WnnE0GTG1yz","status":"notfound","message":"File not found"} No submission file with the given ID was found in any expected location.

• /spool: Stores incoming submissions (managed by docker volumes)

  • /spool/failed: Stores submissions that failed to process
  • /spool/archive: Stores successfully processed submissions

• /state: Stores application state (managed by docker volumes)

  • processed_builds.db: Tracks processed builds
  • processed_tests.db: Tracks processed tests

• /cache: Caches downloaded log files for logspec-worker

docker logs kcidb-rest
docker logs ingester
docker logs logspec-worker
docker logs postgres

docker exec -it postgres psql -U kcidb_editor -d kcidb

The REST API uses JWT for authentication. To disable JWT authentication (not recommended for production):

Uncomment this line in docker-compose.yaml:

#    command: ["/usr/local/bin/kcidb-restd-rs","-j",""]

To manually process a log file through logspec without submitting it to the database, you can run:

docker exec -it logspec-worker python /app/logspec_worker.py --spool-dir /app/spool --origins microsoft --dry-run

This project is licensed under the LGPL-2.1 license.