HAMLET is a holistic and automated framework for evaluating the long-context comprehension of large language models (LLMs).

# Navigate to project directory
cd HAMLET

# Install dependencies
pip install -r requirements.txt

# Edit configuration with your API keys
vim config/config.yaml

# Process a single book
python scripts/run_pipeline.py \
    --input data/books/your_novel.txt \
    --config config/config.yaml

1. Text Chunking: Split novels into manageable 4000-token chunks
2. Tree Extraction: Extract hierarchical key facts (narrative & analytical)
3. Tree Validation: Validate facts across three dimensions
4. Query Generation: Generate targeted queries based on validated trees
5. Summary Generation: Create summaries using multiple language models
6. Summary Evaluation: Evaluate faithfulness, completeness, and conciseness
7. Evaluation Processing: Advanced analysis of evaluation results
8. Metrics Analysis: Generate position-based and comparative metrics