HeadlessX is a self-hosted scraping platform with a web dashboard, protected API, queue-backed workflows, and a remote MCP endpoint.

Current live surfaces:

• Website scraping: scrape, crawl, map, content extraction, screenshots
• Google AI Search
• Tavily
• Exa
• YouTube
• Queue jobs, logs, API keys, proxy management, and config management
• Remote MCP over /mcp

• Simplified the dashboard around one global browser/runtime model
• Added Tavily, Exa, and YouTube workspaces
• Added queued crawl and job flows with Redis + worker support
• Added remote MCP secured with normal dashboard-created API keys
• Added setup and API guides aligned with the current route tree

You can add the HeadlessX CLI skill to AI coding agents such as Cursor, Claude Code, Warp, Windsurf, OpenCode, OpenClaw, Antigravity, and similar tools that support the skills installer flow.

npx skills add https://github.com/saifyxpro/HeadlessX --skill cli

This installs the HeadlessX CLI skill from this repository so the agent can use the published headlessx command and follow the packaged usage guidance.

• Node.js 22+
• pnpm 9+
• PostgreSQL
• Redis
• Python/uv for yt-engine
• Go for the HTML-to-Markdown sidecar

Recommended for most developers:

• PostgreSQL: Supabase or Docker
• Redis: Docker
• App runtime: pnpm dev or mise run dev

This keeps infrastructure simple while still running the app locally.

1. Clone and install:

git clone https://github.com/saifyxpro/HeadlessX.git
cd HeadlessX
pnpm install

2. Create root .env from the full example:

cp .env.example .env

Current root .env.example:

# HeadlessX v2.1.0
# Local development environment

# Database
DATABASE_URL="postgresql://postgres.xxxxx:[email protected]:5432/postgres"

# API server
PORT=8000
HOST=0.0.0.0
NODE_ENV=development

# Required security
# Used by the Next.js dashboard server to authenticate against the API.
DASHBOARD_INTERNAL_API_KEY=replace-with-a-long-random-string

# Used to encrypt stored credentials at rest.
CREDENTIAL_ENCRYPTION_KEY=replace-with-a-different-long-random-string

# Queue and Redis
REDIS_URL=redis://localhost:6379

# Search providers
TAVILY_API_KEY=
EXA_API_KEY=

# Local engines
YT_ENGINE_URL=http://localhost:8090
YT_ENGINE_PORT=8090
YT_ENGINE_TIMEOUT_MS=45000
YT_ENGINE_TEMP_DIR=./tmp/yt-engine
YT_ENGINE_JOB_TTL_HOURS=12

HTML_TO_MARKDOWN_SERVICE_URL=http://localhost:8081
HTML_TO_MARKDOWN_PORT=8081
HTML_TO_MARKDOWN_TIMEOUT_MS=60000

# Browser and stealth defaults are managed in the dashboard settings UI.

# Web dashboard
WEB_PORT=3000
NEXT_PUBLIC_API_URL=http://localhost:8000

# Set this for Docker or custom internal networking.
# INTERNAL_API_URL=http://localhost:8000

# Set this only when the dashboard is hosted on a custom origin.
# FRONTEND_URL=https://dashboard.example.com

If you are using Docker instead of local services, start from the complete Docker env too:

cp infra/docker/.env.example infra/docker/.env

3. Prepare services:

pnpm db:push
pnpm camoufox:fetch

4. Start the workspace:

pnpm dev

This starts:

• web
• api
• worker
• HTML-to-Markdown service
• yt-engine

Important:

• pnpm dev does not provision PostgreSQL or Redis
• Website Crawl requires both Redis and the worker

For the current Docker path:

cp infra/docker/.env.example infra/docker/.env
cd infra/docker
docker compose --profile all up --build -d

Important notes:

• use --profile all
• partial profile runs are not currently reliable because of depends_on relationships
• full Docker now includes yt-engine, so YouTube works inside the same compose stack

See docs/setup-guide.md for the full matrix:

• no-Docker setup
• mixed local setup
• full Docker setup
• MCP client configuration

All non-health backend routes are protected with x-api-key.

Core backend surfaces:

• GET /api/health
• GET/PATCH /api/config
• GET /api/dashboard/stats
• GET /api/logs
• GET/POST/PATCH/DELETE /api/keys
• proxy CRUD under /api/proxies
• website operator routes under /api/operators/website/*
• Google AI Search routes under /api/operators/google/ai-search/*
• Tavily routes under /api/operators/tavily/*
• Exa routes under /api/operators/exa/*
• YouTube routes under /api/operators/youtube/*
• queue job routes under /api/jobs/*
• remote MCP endpoint at /mcp

See the full route reference in docs/api-endpoints.md.

HeadlessX exposes a remote MCP endpoint from the API:

http://localhost:8000/mcp

Use a normal API key created from the dashboard API Keys page.

Do not use DASHBOARD_INTERNAL_API_KEY for MCP clients.

Example client config:

{
  "mcpServers": {
    "headlessx": {
      "transport": "http",
      "url": "http://localhost:8000/mcp",
      "headers": {
        "x-api-key": "hx_your_dashboard_created_key"
      }
    }
  }
}

apps/
  api/                    Express API + worker + MCP
  web/                    Next.js dashboard
  yt-engine/              Python YouTube engine
  go-html-to-md-service/  Go HTML-to-Markdown sidecar
docs/
  setup-guide.md
  api-endpoints.md
infra/docker/

• The dashboard uses the internal dashboard key for server-side internal requests
• MCP uses normal user-created API keys, not the dashboard internal key
• Queue-backed features return degraded/unavailable behavior when Redis is missing
• Docker support now covers the full runtime stack, including yt-engine

See CONTRIBUTING.md for the current contribution workflow, local setup expectations, pull request guidance, and commit message conventions.

MIT