Note
This repo is actively developed. Main branch may be rewritten.
Tools for extracting data from GitHub, storing it in a local SQLite database, querying it, and visualizing trends.
src/ # Rust source (single binary: github-tools)
scripts/util/ # Python: plot.py (charts), json_to_csv.py (utility)
data/ # Committed snapshots: JSON inputs and PNG charts
images/ # Committed PNG charts (promoted from out/images/)
out/ # Gitignored — all generated and local-only files
historical/ # Raw JSON fetched from GitHub API
db/ # SQLite databases
summaries/ # Generated CSVs
images/ # Generated PNG charts (promote to data/images/ to commit)
purge/ # Purge audit logs (local only)
cargo build --release
# Binary: target/release/github-toolsMost commands take an --env-file pointing to a .env file:
GITHUB_TOKEN=...
REPO_OWNER=vectordotdev
REPO_NAME=vector
DOCKER_USERNAME=... # purge commands only
DOCKER_PASSWORD=... # purge commands onlygithub-tools <COMMAND>
Fetch:
fetch-all Fetch issues + discussions for all repos (workflow)
fetch-issues Fetch all issues/PRs for a repository
fetch-discussions Fetch all discussions for a repository
fetch-labels Fetch all labels for a repository
Pipeline:
generate-all Build DB + summaries for all repos (workflow)
build-db Load issues JSON into SQLite database
generate-summaries Generate CSV summaries from SQLite database
Purge:
purge-all Run all purge operations (workflow)
purge nightly Purge old nightly images from GitHub and Docker Hub
purge untagged Purge untagged GitHub container images
purge vector-dev Purge old vector-dev images from Docker Hub
Maintenance:
close-old-prs Close PRs with 'meta: awaiting author' older than 6 months
delete-stale-branches Delete branches with no commits in 4 years
remove-legacy-label Remove legacy type labels from issues/PRs
Run github-tools <COMMAND> --help for full argument details.
github-tools fetch-all --env-file vector.env --env-file vrl.envWrites to out/historical/. Promote to data/ to commit as a snapshot:
cp out/historical/issues/vectordotdev_vector_issues.json data/
cp out/historical/issues/vectordotdev_vrl_issues.json data/
cp out/historical/discussions/vectordotdev_vector_discussions.json data/
cp out/historical/discussions/vectordotdev_vrl_discussions.json data/github-tools generate-all \
--env-file vector.env --env-file vrl.env \
--exclude-labels "no-changelog,meta: awaiting author"
# Charts (still Python):
python scripts/util/plot.py --env-file vector.env --input-dir out/summaries \
--start $(date -d "$(date +%Y-%m-01) -12 months" +%Y-%m) \
--exclude-labels "no-changelog,meta: awaiting author"Promote charts to commit:
cp out/images/*.png data/images/github-tools purge-all --env-file vector.env --dry-run
github-tools purge-all --env-file vector.env # omit --dry-run to executeAudit logs written to out/purge/ (local only).
TODO!
