Burro is a powerful command-line interface (CLI) tool for evaluating Large Language Model (LLM) outputs. It provides both heuristic and LLM-based evaluation methods with secure API key management.

📖 For detailed feature documentation, see FEATURES.md

📊 Heuristic Evaluations (No API key required)

• Levenshtein Distance - Measure string similarity using edit distance
• Exact Match - Perfect matching for IDs, codes, and specific formats
• Case Insensitive Match - Flexible text matching
• Numeric Difference - Compare numerical values with configurable tolerance
• JSON Diff - Analyze structural differences in JSON outputs
• Jaccard Similarity - Calculate similarity between sets of tokens
• Contains - Verify if expected value appears in output

🤖 LLM-as-a-Judge Evaluations (Requires OpenAI API key)

• Factuality - Answer correctness with context validation
• Close QA - Close-ended question matching
• Battle - Compare outputs from different models head-to-head
• Summarization - Evaluate summary quality and accuracy
• SQL - Verify correctness of generated SQL queries
• Translation - Assess translation quality across languages

• 🔒 Secure OpenAI API key management with AES encryption
• 📈 Progress indicators for long-running evaluations
• 💾 Export results to JSON format
• 🎯 Comprehensive error handling and validation
• 🚀 Cross-platform support (Mac, Linux, Windows)
• ⚡ Fast execution with Deno runtime

• For Heuristic Evaluations: None! Works out of the box
• For LLM-based Evaluations: OpenAI API key

sudo curl -L "https://github.com/thisguymartin/burro/releases/download/latest/build-mac-silicon" -o /usr/local/bin/burro && sudo chmod +x /usr/local/bin/burro

sudo curl -L "https://github.com/thisguymartin/burro/releases/download/latest/build-mac-intel" -o /usr/local/bin/burro && sudo chmod +x /usr/local/bin/burro

sudo curl -L "https://github.com/thisguymartin/burro/releases/download/latest/build-linux-arm" -o /usr/local/bin/burro && sudo chmod +x /usr/local/bin/burro

sudo curl -L "https://github.com/thisguymartin/burro/releases/download/latest/build-linux-intel" -o /usr/local/bin/burro && sudo chmod +x /usr/local/bin/burro

1. Download build-windows.exe from the releases page
2. Rename it to burro.exe
3. Move it to your desired location (e.g., C:\Program Files\burro\burro.exe)

burro set-openai-key

Heuristic Evaluation (no API key needed):

burro run-eval -t exact example/exact-match.json

LLM-based Evaluation:

burro run-eval -t factuality example/evals.json

With progress indicators and result export:

burro run-eval -t levenshtein example/levenshtein.json --progress -p

Measures string similarity using edit distance. Great for catching typos and minor variations.

Example: example/levenshtein.json

[
  {
    "input": "Who wrote Hamlet?",
    "output": "William Shakespear",
    "expected": "William Shakespeare"
  }
]

Run:

burro run-eval -t levenshtein example/levenshtein.json

Perfect matching for critical data like IDs, codes, or specific formats.

Example: example/exact-match.json

[
  {
    "input": "What is the ISO code for United States?",
    "output": "US",
    "expected": "US"
  }
]

Run:

burro run-eval -t exact example/exact-match.json

Compare numerical values with configurable tolerance.

Example: example/numeric.json

[
  {
    "input": "What is the value of Pi to 2 decimal places?",
    "output": "3.14",
    "expected": "3.14159",
    "tolerance": 0.01
  }
]

Run:

burro run-eval -t numeric example/numeric.json

Analyze structural differences in JSON outputs.

Example: example/json-diff.json

[
  {
    "input": "Convert user data to JSON",
    "output": "{\"name\": \"John Doe\", \"age\": 30}",
    "expected": "{\"name\": \"John Doe\", \"age\": 30}"
  }
]

Run:

burro run-eval -t json example/json-diff.json

Calculate similarity between sets of tokens.

Example: example/jaccard.json

[
  {
    "input": "List programming languages for web development",
    "output": "JavaScript TypeScript Python Ruby PHP Java",
    "expected": "JavaScript Python Ruby PHP Go"
  }
]

Run:

burro run-eval -t jaccard example/jaccard.json

Evaluate answer correctness with context validation.

Example: example/evals.json

[
  {
    "input": "What is the capital of France?",
    "output": "The capital city of France is Paris",
    "expected": "Paris"
  }
]

Run:

burro run-eval -t factuality example/evals.json

Exact matching for close-ended questions.

Example: example/closeqa.json

[
  {
    "input": "List the first three prime numbers",
    "output": "2,3,5",
    "criteria": "Numbers must be in correct order, separated by commas"
  }
]

Run:

burro run-eval -t closeqa example/closeqa.json

Compare outputs from different models head-to-head.

Example: example/battle.json

[
  {
    "input": "Write a haiku about technology",
    "output": "Code flows like water\nBits and bytes dance in rhythm\nDigital zen speaks",
    "expected": "Silicon pathways\nData streams through endless night\nMachines dream in code"
  }
]

Run:

burro run-eval -t battle example/battle.json

Evaluate the quality and accuracy of text summaries.

Example: example/summarization.json

[
  {
    "input": "Summarize this text",
    "output": "Climate change impacts polar regions; urgent global action needed.",
    "context": "Long article about climate change effects..."
  }
]

Run:

burro run-eval -t summarization example/summarization.json

Verify the correctness of generated SQL queries.

Example: example/sql.json

[
  {
    "input": "Find all users over age 18",
    "output": "SELECT * FROM users WHERE age > 18;",
    "expected": "SELECT * FROM users WHERE age > 18;",
    "context": "Database schema: users(id, name, email, age)"
  }
]

Run:

burro run-eval -t sql example/sql.json

Assess translation quality across languages.

Example: example/translation.json

[
  {
    "input": "Translate 'Hello' to Spanish",
    "output": "Hola",
    "expected": "Hola"
  }
]

Run:

burro run-eval -t translation example/translation.json

See SCENARIOS.md for comprehensive real-world examples including:

• 🎯 Customer Support Bot Evaluation
• 💻 Code Generation Validation
• 🌍 Translation Quality Assessment
• ⚔️ Chatbot Response Comparison
• 📊 Data Extraction Accuracy
• 📚 Educational Content Assessment

• AES-256 Encryption for API key storage
• Secure key generation using Web Crypto API
• Encrypted SQLite storage for settings
• No plaintext secrets ever stored on disk

For long-running evaluations:

burro run-eval -t factuality large-dataset.json --progress

Save evaluation results to JSON:

burro run-eval -t battle comparison.json -p
# Results saved to ~/Downloads/comparison.json-result.json

Run multiple evaluations sequentially:

burro run-eval -t exact tests/ids.json
burro run-eval -t factuality tests/qa.json
burro run-eval -t sql tests/queries.json

To determine which version to download:

uname -m

• arm64 → Use Apple Silicon version (M1/M2/M3)
• x86_64 → Use Intel version

uname -m

• aarch64 or arm64 → Use ARM version
• x86_64 → Use Intel version

sudo chmod +x /usr/local/bin/burro

1. Verify installation location is in your PATH
2. Restart your terminal
3. Check executable exists: ls -l /usr/local/bin/burro

burro set-openai-key  # Re-enter your API key

• Exact match: Check for extra spaces or case differences
• Numeric: Adjust tolerance values
• JSON: Ensure consistent formatting
• Factuality: Make expected answers less specific

sudo rm /usr/local/bin/burro
which burro  # Should return nothing

1. Delete burro.exe from installation location
2. Remove from PATH if added

All evaluation types have examples in the /example directory:

example/
├── closeqa.json           # Close-ended QA
├── evals.json            # Factuality evaluation
├── levenshtein.json      # String similarity
├── exact-match.json      # Exact matching
├── numeric.json          # Numeric comparison
├── json-diff.json        # JSON structure diff
├── jaccard.json          # Token similarity
├── battle.json           # Model comparison
├── summarization.json    # Summary quality
├── sql.json             # SQL validation
└── translation.json      # Translation quality

1. Install Burro using the command for your platform
2. Try a heuristic evaluation (no API key needed):

   burro run-eval -t exact example/exact-match.json

3. Set up your API key for LLM evaluations:

   burro set-openai-key

4. Run an LLM evaluation:

   burro run-eval -t factuality example/evals.json

5. Explore scenarios in SCENARIOS.md
6. Create your own evaluation files based on examples

1. Start with heuristics - They're fast and free
2. Use the right tool - Match evaluation type to your use case
3. Build incrementally - Start with 5-10 test cases
4. Version your tests - Track evaluation files in git
5. Automate regularly - Run evaluations as part of your workflow
6. Compare methods - Try multiple evaluation types on the same data
7. Check examples - Learn from provided examples in /example directory

• FEATURES.md - Complete feature documentation and technical details
• SCENARIOS.md - Real-world use cases and examples
• /example - Sample evaluation files for all types
• README.md - This file, quick start guide and overview

Contributions welcome! Please feel free to submit a Pull Request.

MIT License - feel free to use Burro in your projects!

Ready to evaluate your LLM outputs?

1. Install Burro for your platform
2. Pick an evaluation type that matches your needs
3. Create or use an example evaluation file
4. Run your first evaluation
5. Iterate and improve!

Need help? Check the issues page or review the examples!

Made with ❤️ for LLM developers and evaluators