Turn any website into a CLI/API for AI agents.

AI agents interact with websites through browser automation, which is slow, expensive, and unreliable:

site2cli uses Progressive Formalization — a 3-tier system that automatically graduates interactions from slow-but-universal to fast-but-specific:

graph LR
    A["Tier 1: Browser<br/>Exploration"] -->|"Pattern<br/>detected"| B["Tier 2: Cached<br/>Workflow"]
    B -->|"API<br/>discovered"| C["Tier 3: Direct<br/>API Call"]
    style A fill:#ff6b6b,color:#fff
    style B fill:#ffd93d,color:#000
    style C fill:#6bcb77,color:#fff

The Discovery Pipeline captures browser traffic and converts it into structured interfaces:

graph TD
    A[Launch Browser + CDP] --> B[Capture Network Traffic]
    B --> C[Group by Endpoint Pattern]
    C --> D[LLM-Assisted Analysis]
    D --> E[OpenAPI 3.1 Spec]
    E --> F[Python Client]
    E --> G[CLI Commands]
    E --> H[MCP Server]

# Install (lightweight - no browser deps by default)
pip install site2cli

# Install with all features
pip install site2cli[all]

# Or pick what you need
pip install site2cli[browser]   # Playwright for traffic capture
pip install site2cli[llm]       # Claude API for smart analysis
pip install site2cli[mcp]       # MCP server generation

# Capture traffic and discover API endpoints
site2cli discover kayak.com --action "search flights"

# site2cli launches a browser, captures network traffic,
# and generates: OpenAPI spec + Python client + MCP tools

# CLI
site2cli run kayak.com search_flights from=SFO to=JFK date=2025-04-01

# Or as MCP tools for AI agents
site2cli mcp generate kayak.com
site2cli mcp serve kayak.com

from site2cli.discovery.analyzer import TrafficAnalyzer
from site2cli.discovery.spec_generator import generate_openapi_spec
from site2cli.generators.mcp_gen import generate_mcp_server_code

# Analyze captured traffic
analyzer = TrafficAnalyzer(exchanges)
endpoints = analyzer.extract_endpoints()

# Generate OpenAPI spec
spec = generate_openapi_spec(api)

# Generate MCP server
mcp_code = generate_mcp_server_code(site, spec)

From a single discovery session, site2cli produces:

graph TB
    subgraph "Interface Layer"
        CLI[CLI - Typer]
        MCP[MCP Server]
        SDK[Python SDK]
    end
    subgraph "Router"
        R[Tier Router + Fallback]
    end
    subgraph "Execution Tiers"
        T1[Tier 1: Browser]
        T2[Tier 2: Workflow]
        T3[Tier 3: API]
    end
    subgraph "Discovery Engine"
        CAP[Traffic Capture - CDP]
        ANA[Pattern Analyzer]
        GEN[Code Generators]
    end
    CLI --> R
    MCP --> R
    SDK --> R
    R --> T1
    R --> T2
    R --> T3
    CAP --> ANA --> GEN

site2cli has been validated with 7 experiments across 15+ real public APIs — a comprehensive pre-launch test suite:

All 7 experiments pass in ~74 seconds.

# Auto-generated client for JSONPlaceholder — no human code
client = JSONPlaceholderClient()
albums = client.get_albums()
# → [{"userId": 1, "id": 1, "title": "quidem molestiae enim"}, ...]

# Auto-generated client for Open-Meteo — handles query params
client = OpenMeteoClient()
weather = client.get_v1_forecast(latitude="37.77", longitude="-122.42", current_weather="true")
# → {"current_weather": {"temperature": 12.3, "windspeed": 8.2, ...}}

Reproduce all experiments: python experiments/run_all_experiments.py

214 tests (208 unit/integration + 6 live), all passing on Python 3.10+.

# Clone and install with dev dependencies
git clone https://github.com/lonexreb/site2cli.git
cd site2cli
pip install -e ".[dev]"

# Run tests
pytest                         # Unit + integration tests (no network)
pytest -m live                 # Live tests (hits real APIs)
pytest -v                      # Verbose output

# Lint
ruff check src/ tests/

• Anthropic API key (ANTHROPIC_API_KEY): Used for LLM-assisted endpoint analysis. Optional — discovery works without it, just without enhanced descriptions.
• No other keys required for core functionality.

• Cookie banner auto-dismissal — 3-strategy detection (30+ vendor selectors, multilingual text matching, a11y role matching) runs automatically during discovery
• Auth page detection — Detects login/SSO/OAuth/MFA/CAPTCHA pages and suggests site2cli auth login
• Accessibility tree extraction — Better page representation for LLM-driven exploration (replaces CSS-only element extraction)
• Action retry logic — Configurable retries with delay for click/fill/select/press actions
• Rich wait conditions — 9 condition types: network-idle, load, exists:<selector>, visible:<selector>, hidden:<selector>, url-contains:<text>, text-contains:<text>, stable
• Output filtering — --grep, --limit, --keys-only, --compact flags on site2cli run
• Agent init command — site2cli init generates Claude MCP config or generic agent prompts from discovered sites
• 214 tests (up from 156), all passing

• Core discovery pipeline (traffic capture → OpenAPI → client)
• MCP server generation
• Community spec sharing (export/import)
• Health monitoring and self-healing
• Tier auto-promotion (Browser → Workflow → API)
• PyPI package publication
• Pre-launch validation suite (7 experiments, 15+ APIs, all passing)
• Cookie banner handling & auth page detection
• Accessibility tree extraction for browser exploration
• Agent init/config generation
• Output filtering for run results
• OAuth device flow support
• Workflow recording UI
• Multi-site orchestration
• Trained endpoint classifier (replace heuristics)

MIT