FIA Docs Scraper

This Application scrapes the FIA Website for new Events and Documents, it downloads the Documents and uploads them onto an S3 bucket, it also makes screenshots of every page of the Document and uploads them onto S3 as well.

Written in Rust and fully dockerized for easy deployment.

Setup

Configuration

This application will be configured using Environment Variables. If available the app will try to load a .env file.

Here is an example .env:

DATABASE_TOKEN=<Libsql JWT>
S3_ACCESS_KEY=<S3 Bucket Access Key>
S3_SECRET_KEY=<S3 Bucket Secret Key>
SENTRY_DSN=<Sentry DSN for Error reporting>

Docker

This App is generally speaking used together with the FIA-Docs-Bot.

Here is a docker-compose.yaml that encompasses

• A Libsql Database
• This FIA-Docs-Scraper
• The FIA-Docs-Bot

services:
  libsql-database:
    image: ghcr.io/tursodatabase/libsql-server:latest
    volumes:
        - ./data/libsql:/var/lib/sqld
    container_name: database
    env_file: ./.db.env
  discord-bot:
    image: ghcr.io/markustheort/fia-docs-bot:latest
    container_name: fia-docs-bot
    restart: unless-stopped
    env_file: ./.bot.env
    environment:
      - DATABASE_URL=http://database:8080
    depends_on:
     - libsql-database
  docs-scraper:
    image: codeberg.org/mto/fia-docs-scraper:latest
    container_name: fia-docs-scraper
    restart: unless-stopped
    env_file: ./.scraper.env
    environment:
      - DATABASE_URL=http://database:8080
    depends_on:
      - libsql-database
    stop_grace_period: 30s

Technologies used

• Rust with Tokio for async runtime.
• Imagemagick for Document screenshots
• S3-Compatible Storage for screenshot and mirror host.
• Libsql (SQLite) for Data storage.
• Sentry for error reporting.

License

This project is licensed under the MIT or Apache License. (see MIT License, see Apache License)