The Biodiversity Cell Atlas is a coordinated international effort aimed at molecularly characterizing cell types across the eukaryotic tree of life. Our mission is to pave the way for the efficient expansion of cell atlases to hundreds of species.

This project uses:

• Podman Compose to manage multiple Podman containers (using docker-compose backend for compatibility)
• Ghost, a blog-focused Content Management System (CMS) for the main website
  • Mailpit provides a web interface to read Ghost transactional emails
• Django, a high-level Python web framework powering the data portal, with additional dependencies (see Dockerfile)
  • Bun to build JavaScript and CSS assets from external libraries
  • DIAMOND for fast sequence alignment
  • Gunicorn to serve the Django app in production
• PostgreSQL, a relational database
• Nginx, a reverse proxy

The project configuration is defined in compose.yml.

To set up the project and run the web app locally, first install:

• Podman — consider installing via Podman Desktop to make it easier to manage Podman containers
• docker-compose (standalone) — Docker itself is not required

Then, download the project directory from GitHub and follow these steps:

# Go to the project directory
cd bca-website

# Run setup.sh to setup configuration files from *.template files (edit them afterwards as needed):
# env.template, nginx/nginx.conf, .pg_service.conf, .pgpass
./scripts/setup.sh

# Start Podman Compose to locally deploy the web app
# - Prepares, downloads and starts all containers
# - `-d`: starts the containers in detached mode
# - `--build`: rebuilds the web image (for instance, new Python dependencies in `requirements.txt`)
podman compose up -d --build

# Create a superuser (only required once for database setup)
podman compose exec web python manage.py createsuperuser

These are some of the commands to use during development:

# Locally deploy the web app to localhost
podman compose up -d --build

# Check information about the active Compose containers
podman compose ps

# Check container logs
# - use `-f` to live update log output
# - add container name to print logs only for that container
podman compose logs
podman compose logs -f
podman compose logs web

# Run a bash shell within the web app container
podman compose exec web bash

# Run a Python shell within the context of the web app
# https://docs.djangoproject.com/en/dev/intro/tutorial02/#playing-with-the-api
podman compose exec web python manage.py shell

# Run unit tests: https://docs.djangoproject.com/en/dev/topics/testing/
podman compose exec web python manage.py test

# Stop and delete all containers and Compose-related networks
podman compose down

The project directory is automatically mounted to the web app container, allowing to preview updates in the web app in real-time, except for static files and Django model updates.

After launching the service, the main website will be deployed to http://localhost and the Data Portal to http://portal.localhost.

A dedicated Compose file (such as compose.prod.yml) can be used for production-specific settings:

# Set COMPOSE_FILE in .env: COMPOSE_FILE=compose.yml:compose.prod.yml
# Deploy in production mode
podman compose -d

The main website is built with the Ghost blogging platform. Base templates in the ghost/ folder modify the default theme.

Transactional emails (like those sent to reset passwords and create new user accounts) can be read by opening Mailpit web interface at localhost:1025.

The Data Portal is powered by Django and its image is built from Dockerfile. The latest images are available on GitHub Packages.

The Data Portal is organized into two directories:

• app contains the models and templates for the Data Portal
• rest contains the REST API code and its documentation

To apply changes to Django models, run the migrate command:

podman compose exec web python manage.py migrate

The migrate command runs automatically when the web app container starts in development mode, so you can simply run podman compose restart web. The automatic command will not work if there is an issue that requires manual intervention.

When you start the Compose project (podman compose up web), entrypoint.sh runs Bun to build JavaScript and CSS assets from TypeScript and external libraries, then runs Django to collect the static files. Nginx automatically serves the collected files from the output folder.

To manually update static files, you can also run these commands:

# Bun: install JS and CSS dependencies
podman compose exec web bun install

# Bun: build custom and third-party JS and CSS assets
podman compose exec web bun run build

# Django: collect all static files
podman compose exec web python manage.py collectstatic --noinput

Run all Django unit tests to check the app's functionality:

podman compose exec web python manage.py test

You can verify the deployment configuration:

podman compose exec web python manage.py check-deploy

By default, the Django app uses the Postgres database service in compose.yml. However, you can instead connect to any database by editing the Postgres files .pg_service.conf and .pgpass, and then changing to which database service to connect in .env:

POSTGRES_SERVICE=remote-bca-db

In case the database service is not needed because you are connecting to an external database, edit the .env file to exclude the db profile:

# Change the following line to exclude the db service

# COMPOSE_PROFILES=nginx,db
COMPOSE_PROFILES=nginx

If the database can only be accessed via an intermediate host, you will need to connect to the host via an SSH tunnel:

ssh -fN -L 5432:db-host.com:5432 [email protected]

To connect to the database through the SSH tunnel, use host host.docker.internal. You can configure your .pg_service.conf and .pgpass files like this:

[ssh-bca-db]
host=host.docker.internal
port=5432
dbname=bca_db
user=wallace

host.docker.internal:5432:bca_db:wallace:mypassword

You can now start the project as usual via podman compose up.

To check the Nginx configuration that is going to be run:

podman compose exec nginx nginx -t

In case you want to disable the Nginx service, edit the .env file to exclude the nginx profile:

# Change the following line to exclude the nginx service

# COMPOSE_PROFILES=nginx,db
COMPOSE_PROFILES=db

# If you don't need both the nginx and db services, simply delete the whole line

Unit tests are automatically run using Django and Bun for every pull request using GitHub Actions. Their coverage reports are then uploaded to Codecov.

The tests and coverage reports can also be manually run with the following commands.

Run all Django unit tests with:

# Locally deploy the web app
podman compose up -d --build

# Run Django tests and report coverage in HTML (open the HTML file with a web browser)
podman compose exec web coverage run manage.py test
podman compose exec web coverage html

Run all TypeScript tests with Bun:

# Locally deploy the web app
podman compose up -d --build

# Run Bun tests and report coverage as text in the terminal
podman compose exec web bun test --coverage

# Run Bun tests in watch mode to automatically re-run tests on file changes
podman compose exec web bun test --watch

Check and lint Django templates using djlint:

# Locally deploy the web app
podman compose up -d --build

# Show errors in Django  template files
podman compose exec web djlint .

# Automatically lint and reformat Django template files
podman compose exec web djlint . --reformat

Process CSS with Autoprefixer to add vendor prefixes:

# Locally deploy the web app
podman compose up -d --build

# Process all CSS files and replace them in-place
podman compose exec web bunx postcss ./**/*.css --use autoprefixer --no-map --replace

Super-Linter is run for every Pull Request. To run it locally using Podman, execute the following commands (the correct image is automatically pulled based on the version used in the GitHub workflow):

# Run in check mode on changed files
./superlinter.sh check

# Run in fix mode on changed files
./superlinter.sh fix

# Run in fix mode on changed files using Python and JS linters only
./superlinter.sh fix --python --js

# Run in fix mode on all codebase
./superlinter.sh fix --all

# Print all available options
./superlinter.sh

The environment files that Super-Linter automatically loads are available in .github/linters: super-linter.env and super-linter-fix.env.