An automated monitoring service that queries Tempo for high-latency traces and creates GitHub issues with detailed performance reports. Runs on a scheduled basis to help track and resolve performance bottlenecks in your distributed systems.

• 🔍 Automatic Trace Detection - Queries Tempo for traces exceeding 500ms
• 📊 Smart Deduplication - Groups traces by root name and reports only the slowest instance
• 🐙 GitHub Integration - Automatically creates detailed issues with trace analysis
• ⏰ Scheduled Reports - Runs daily at midnight UTC (configurable)
• 🚀 FastAPI Server - REST API with health checks and manual trigger endpoint
• 🐳 Docker Ready - Easy deployment with Docker and Docker Compose

• Python 3.13+ (or use Docker)
• Access to a Tempo instance
• GitHub Personal Access Token with repo permissions
• Docker & Docker Compose (for containerized deployment)

1. Clone the repository

   git clone <your-repo-url>
   cd <repo-directory>

2. Configure environment variables

   Create a .env file in the project root:

   # Tempo Configuration
   TEMPO_BASE=https://tempo.your-domain.com
   
   # GitHub Configuration
   GITHUB_TOKEN=ghp_your_personal_access_token_here
   GITHUB_REPO=username/repository-name
   
   # Server Port (optional, defaults to 8070)
   PORT=8070

3. Build and run

   docker-compose up -d

4. Check logs

   docker-compose logs -f reporter

# Build the image
docker build -t latency-reporter:latest .

# Run the container
./run.sh
# or manually:
docker run -d \
  --name latency-reporter \
  -p 8000:8000 \
  --env-file .env \
  --restart unless-stopped \
  latency-reporter:latest

1. Create virtual environment

   python3 -m venv venv
   source venv/bin/activate  # On Windows: venv\Scripts\activate

2. Install dependencies

   pip install -r requirements.txt

3. Set environment variables

   export TEMPO_BASE=https://tempo.your-domain.com
   export GITHUB_TOKEN=ghp_your_token_here
   export GITHUB_REPO=username/repo

4. Run the application

   uvicorn main:app --host 0.0.0.0 --port 8070

Edit the configuration section in main.py:

# Trace query settings
USE_TRACEQL = True
TRACEQL_QUERY = '{ duration > 500ms }'     # Adjust threshold
LIMIT = 100                                 # Max traces to fetch

# Thresholds
MIN_TRACES_TO_CREATE_ISSUE = 1             # Minimum traces before creating issue
MAX_DURATION_TO_HIGHLIGHT_MS = 2500        # Mark traces above this as "very slow"

# GitHub labels
GITHUB_LABELS = ["performance", "latency", "tempo"]

GET /health

Returns server health status.

Response:

{
  "status": "healthy"
}

POST /trigger

Manually triggers the latency report generation (bypasses schedule).

Response:

{
  "message": "Latency report triggered successfully."
}

Example:

curl -X POST http://localhost:8070/trigger

1. Query Tempo - Fetches traces from the last 72 hours matching the TraceQL query
2. Deduplicate - Groups traces by rootTraceName and keeps only the slowest instance
3. Analyze - Sorts traces by duration and formats detailed information
4. Create Issue - Posts a GitHub issue with:
   • Trace IDs and durations
   • Root service names
   • Timestamps
   • Direct links to Grafana for trace visualization
   • Tags and metadata

The default schedule runs at midnight UTC daily. To customize:

Edit the cron trigger in main.py:

@app.on_event("startup")
def startup_event():
    # Daily at midnight UTC
    scheduler.add_job(run_report, CronTrigger(hour=0, minute=0))
    
    # Every 6 hours
    # scheduler.add_job(run_report, CronTrigger(hour='*/6'))
    
    # Every Monday at 9 AM
    # scheduler.add_job(run_report, CronTrigger(day_of_week='mon', hour=9))
    
    scheduler.start()

1. Go to GitHub Settings → Developer settings → Personal access tokens
2. Generate a new token (classic) with repo scope
3. Copy the token and add it to your .env file
4. Never commit the token to version control

The generated issues include direct links to view traces in Grafana. Ensure:

• Your Grafana instance is accessible
• The Tempo datasource is configured
• Update the Grafana URL in main.py if needed:

grafana_link = (
    f"https://grafana.your-domain.com/explore?"
    f"orgId=1&..."
)

The service includes a health check that:

• Runs every 30 seconds
• Checks http://localhost:8070/health
• Retries 3 times with 10s timeout
• Waits 10s after startup before checking

Configured with JSON file driver:

• Max file size: 10MB
• Max files: 3 (30MB total)

• Verify TEMPO_BASE is correct and accessible
• Check if traces actually exceed the duration threshold (500ms)
• Increase LIMIT if you have many traces
• Verify the time window (currently 72 hours)

• Verify GITHUB_TOKEN has correct permissions
• Check GITHUB_REPO format: owner/repository
• Ensure token hasn't expired
• Check GitHub API rate limits

• Verify .env file exists and has all required variables
• Check port 8070 isn't already in use
• Review container logs: docker-compose logs reporter

• Check scheduler logs in container output
• Verify timezone settings (scheduler uses UTC)
• Test with manual trigger: POST /trigger

# Add tests in tests/ directory
pytest

# Format code
black main.py

# Lint
flake8 main.py

MIT

Contributions are welcome! Please:

1. Fork the repository
2. Create a feature branch
3. Commit your changes
4. Push to the branch
5. Open a Pull Request

For issues and questions:

• Open a GitHub issue
• Check existing issues for solutions
• Review the troubleshooting section

Generated by: High Latency Reporter
Last Updated: February 2026