Documentation package registry for ContextQMD. A Rails application that crawls, indexes, and serves library documentation for MCP-based coding assistants.

The registry is the backend for contextqmd-mcp. It:

• Crawls documentation from GitHub, GitLab, Bitbucket repos, llms.txt files, websites, and OpenAPI specs
• Splits docs into versioned pages with headings, checksums, and full-text search metadata
• Builds deterministic tar.gz bundles per library version for bundle-first installs
• Serves a public REST API consumed by MCP clients
• Provides a web UI for browsing libraries, submitting crawl requests, and managing accounts
• Includes a staff admin panel for library/user/proxy management

bin/setup              # Install deps, create databases, run migrations, seed
bin/dev                # Start Rails + Vite dev server + Solid Queue

Requires PostgreSQL running locally. Default config connects to localhost:5432.

Rails uses 5 databases per environment: primary, cache (Solid Cache), queue (Solid Queue), cable (Solid Cable), and analytics.

To wipe stale local docs data and queue a fresh crawl set:

bin/rake contextqmd:dev:refresh_catalog

This queues a small canonical catalog (rails/rails, inertia-rails, vite_ruby, react.dev, next.js, expressjs.com) using the first staff identity. Override with:

[email protected] \
URLS='https://github.com/rails/rails,https://github.com/inertiajs/inertia-rails' \
bin/rake contextqmd:dev:submit_catalog

Identity ──< User >── Account
    │                    │
    ├──< AccessToken     ├──< Library ──< Version ──< Page
    │                    │       │           ├──< Bundle (tar.gz archives)
    ├──< CrawlRequest ──┘       │           └── FetchRecipe
                                └── SourcePolicy

CrawlProxyConfig ──< CrawlProxyLease

• Identity — authentication entity (Devise). One identity can belong to multiple accounts (multi-tenancy).
• Account — tenant boundary. Has an owner user and a system user.
• Library — a documentation package (e.g., react, rails). Has JSON aliases for resolution and crawl rules.
• Version — belongs to a library (e.g., 18.2.0, latest). Channels: stable, latest, canary, snapshot.
• Page — a single documentation page. Markdown content stored in description column. Full-text search indexed.
• Bundle — downloadable tar.gz archive of all pages for a version. Backed by Active Storage (R2 or local).
• FetchRecipe — records how a version was crawled (source type, URL, normalizer config).
• SourcePolicy — licensing and mirroring permissions for a library.
• CrawlRequest — job entry point. On creation, enqueues ProcessCrawlRequestJob for the full fetch-and-import pipeline.
• CrawlProxyConfig/Lease — proxy pool for outbound HTTP during crawling, with health tracking and lease-based concurrency.

All read endpoints are public (no auth). Write endpoints use Bearer token authentication. Rate limits: 300/min default, 120/min for resolve, 60/min for query.

1. POST /api/v1/resolve — resolve a query to the canonical library slug + version
2. GET /api/v1/libraries/:slug/versions/:v/manifest — get version manifest with page checksums
3. GET /api/v1/libraries/:slug/versions/:v/bundles/:profile — download docs bundle

Source type is auto-detected from the URL. Submit crawl requests through the web UI or the API.

The pipeline supports configurable crawl rules per library (include/exclude paths), a proxy pool with health tracking and exponential backoff, and SSRF protection for all outbound requests.

The frontend is a React 19 + Inertia.js SPA with three layout zones:

• Home — hero search, tabbed library table (popular/trending/recent), feature cards
• Libraries — browse/search with card grid, library detail with pages/versions/usage tabs
• Rankings — libraries ranked by documentation coverage
• Queue — live crawl request status (active/completed tabs)
• Doc pages — full page view with TOC sidebar and markdown rendering

• Dashboard — account stats, recent crawls and libraries
• Submit docs — crawl request form with source type selector
• Access tokens — create/revoke API tokens
• Settings — profile, password, account cancellation

• Dashboard — system-wide stats and recent crawl activity
• Libraries — CRUD, version management, re-crawl, crawl rules editor
• Users — identity management, suspend/unsuspend, staff access, bulk actions
• Proxy pool — CRUD for crawl proxy configs with health/capacity monitoring
• Jobs — Mission Control dashboard for Solid Queue

Queue config: default (3 threads), crawl_website (1 thread, isolated for slow BFS crawls). Recurring: hourly Solid Queue cleanup, daily account incineration at 3am.

bin/rails test             # All tests
bin/rubocop                # Ruby linting
npm run check              # TypeScript type check
npm run lint               # ESLint
npm run format:check       # Prettier
bin/ci                     # Full CI pipeline (all of the above + security scans + asset build)

CI runs on GitHub Actions with 3 parallel jobs: security scan (Brakeman + bundler-audit), lint (RuboCop + TS + ESLint + Prettier), and test (PostgreSQL service container).

app/
  controllers/
    api/v1/              # Public REST API (ActionController::API)
    app/                 # Authenticated web app (account-scoped)
    admin/               # Staff-only admin panel
    concerns/            # Auth, authorization, error handling, CSRF, tenanting
    identities/          # Devise session/registration/password/OAuth controllers
  models/
    account/             # Cancellable, Incineratable concerns
    user/                # Named, Role concerns
    docs_fetcher/        # Crawl pipeline (Git, LlmsTxt, Website, Openapi)
    docs_fetcher/git/    # GitHub, GitLab, Bitbucket fetchers
    docs_fetcher/website/ # RubyRunner (BFS), NodeRunner (Playwright)
  jobs/                  # ProcessCrawlRequestJob, BuildBundleJob, AccountIncinerationJob
  frontend/
    pages/               # 41 Inertia React pages (public, app, admin, auth, errors)
    components/          # Shared, admin, app components + shadcn/ui primitives
    layouts/             # PersistentLayout, PublicLayout, AdminLayout, AppLayout
    lib/                 # Utilities (inertia resolver, account scoping, date formatting)
    types/               # TypeScript type definitions
    entrypoints/         # Vite entry points (client + SSR)
config/
  deploy.yml             # Kamal deployment (3 containers: web, vite SSR, jobs)
  storage.yml            # Local disk + Cloudflare R2 (public/private)
  queue.yml              # Solid Queue worker pools
  recurring.yml          # Recurring jobs (cleanup, incineration)
test/
  models/                # 16+ model test files including all fetcher types
  controllers/api/v1/    # API endpoint tests
  integration/           # Auth, tenanting, scoping integration tests
  jobs/                  # Job tests
  fixtures/              # YAML fixture data

Deployed via Kamal to a single Hetzner server running 3 Docker containers:

• web — Rails + Thruster/Puma (port 80, Let's Encrypt SSL)
• vite — SSR server (node public/vite-ssr/ssr.js)
• job — Solid Queue worker

PostgreSQL 18 runs as a Kamal accessory on the same host. Active Storage files persist via a Docker volume, with Cloudflare R2 for production bundle storage.

MIT