documents to RAG — Drag any docs into your AI workflow.

Model Context Protocol (MCP) server for crawling documentation websites and enabling semantic search via Claude Code.

Index any documentation site and search it using natural language — just drag it in.

• Web Crawling: Automatically crawl and index documentation sites with configurable depth
• Hybrid Search: Combines semantic search with BM25 keyword matching using Reciprocal Rank Fusion
• Semantic Chunking: Intelligent text splitting that preserves paragraph boundaries
• Query Expansion: Optional query variations for improved recall
• Local Embeddings: Uses Ollama with nomic-embed-text (runs locally, no API keys needed)
• Persistent Storage: ChromaDB for reliable vector storage
• Security: SSRF protection, rate limiting, and input validation built-in

• Docker and Docker Compose
• Claude Code CLI
• 16GB RAM recommended (for embedding model)

git clone https://github.com/your-username/drag-mcp.git
cd drag-mcp

docker compose up -d ollama chroma

docker compose exec ollama ollama pull nomic-embed-text

This downloads the nomic-embed-text model (~274MB). Only needed once.

Add the following to your Claude Code settings (~/.claude/settings.json):

{
  "mcpServers": {
    "drag": {
      "command": "docker",
      "args": [
        "compose", "-f", "/path/to/drag-mcp/docker-compose.yml",
        "run", "--rm", "-i", "mcp-server"
      ]
    }
  }
}

Replace /path/to/drag-mcp with the actual path to your cloned repository.

The MCP server will now be available in Claude Code.

Use natural language in Claude Code to interact with the server:

Index a documentation site:

"Index https://docs.example.com with depth 3"

Search indexed documents:

"Search for authentication in the docs"

List indexed sources:

"Show me all indexed documentation sources"

Delete a source:

"Remove the docs from example.com"

• hybrid (default: true): Enable hybrid search combining semantic similarity with keyword matching
• expand_query (default: false): Generate query variations for improved recall
• top_k (default: 5, max: 100): Number of results to return

┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐
│   Claude Code   │────▶│   MCP Server    │────▶│     Ollama      │
│                 │     │   (TypeScript)  │     │ (nomic-embed)   │
└─────────────────┘     └────────┬────────┘     └─────────────────┘
                                 │
                                 ▼
                        ┌─────────────────┐
                        │    ChromaDB     │
                        │  (Vector Store) │
                        └─────────────────┘

npm install
npm run build

# Start dependencies
docker compose up -d ollama chroma

# Run in development mode
npm run dev

src/
├── index.ts           # MCP server entry point
├── config.ts          # Centralized configuration
├── logger.ts          # Structured logging
├── types.ts           # Shared type definitions
├── services/
│   ├── crawler.ts     # Web crawler with SSRF protection
│   ├── chunker.ts     # Semantic text chunking
│   ├── embedder.ts    # Ollama embedding client
│   └── vectorstore.ts # ChromaDB operations & hybrid search
└── tools/
    ├── crawl.ts       # crawl_and_index tool
    ├── search.ts      # search_docs tool
    ├── list.ts        # list_sources tool
    └── delete.ts      # delete_source tool

This server includes several security measures:

• SSRF Protection: Blocks requests to internal networks (localhost, private IPs, cloud metadata endpoints)
• Rate Limiting: 500ms delay between crawl requests to avoid overwhelming target servers
• Input Validation: URL validation, max depth limits, query length limits
• Response Size Limits: 10MB max per page to prevent memory exhaustion

Ensure Ollama is running and the model is pulled:

docker compose exec ollama ollama list
# Should show: nomic-embed-text

• Check if the URL is accessible
• Some sites may block crawlers; check the target site's robots.txt
• Try a simpler URL first (e.g., a single page instead of the root)

Ensure Docker services are running:

docker compose ps
# All services should show "Up"

Contributions are welcome! Please:

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

Please ensure your code passes linting and tests:

npm run lint
npm run format:check
npm test

This project is licensed under the MIT License - see the LICENSE file for details.

• Model Context Protocol by Anthropic
• Ollama for local LLM inference
• ChromaDB for vector storage
• nomic-embed-text for embeddings