Overview

This PR implements a attestation-based CI system moves a bunch of the work to before you git push and also caches results for speed.

The goal is that we could get PRs as far as the "ready to merge" state before having to run a million hours in CI.

• Git hooks run the checks and cache the results. pre-commit just warns if there are failures, pre-push will block a push by default if it would fail CI, but you can override it if you just want to save your stuff to Github.
• on feature branches: CI only verifies proofs exist (<10 seconds)
• on trunk: Full tests still run on all architectures, but transcripts don't (tbd)

./scripts/check.sh, which is the one-stop shop for checks, now uses cached test results if available.

Usage

Git hooks

Install the git hooks with ./scripts/hooks/install.bash, and that's it.

Manual way

1. Run ./scripts/check.sh (because it runs the tests and the transcripts) or the individual ./scripts/proofs/*.sh
2. Check in: .github/workflows/proofs/*

Implementation approach and notes

• Version-controlled proofs vs local (untracked) proofs
  • Local proofs (.gitignored in .local-proofs) keep the last 100 checks around for reuse; dropping the least recently added.
  • Version-controlled proofs file normally contains just the single current hash for CI to verify, git attribute merge=union to avoid always having a merge conflict
• Each check has file dependencies, listed with glob patterns in the respective scripts/proofs/*.sh script.
  • The "code" files (.yaml, .hs, .md) vs the "infrastructure" files (scripts/proofs/*.sh) are hashed separately, so that the latter can be a warning.
• ci.yaml has a verify-proofs job on feature branches instead of running actual tests.
• a bunch of tests (all?) still run in the merge queue to catch stuff that may have slipped through.
• transcript tests don't get run on trunk, they become more about checking for output changes than testing; although it's debatable.

Interesting/controversial decisions

• pre-commit caches results but doesn't abort on failure
• pre-push aborts of tests or transcripts fail unless --no-verify
• keep last 100 checks locally. using just one file would be pretty okay, but this catches some places where the cache would needlessly be invalidated otherwise
• when do we check and update transcripts on non-FF merges? we just check them in merge queues; if they need updating, then you merge in trunk yourself into the PR and hope for the best?

Test coverage

tested these cases manually:

• attestation in local file but not in shared file: skips tests and writes shared file
• attestation in shared file but not in local file: adds it to local file
• modifying a key file invalidates attestation
• modifying a non-key file doesn't invalidate attestation
• ci fails if key file modified

Loose ends

• want to make transcripts faster to run in general, whether by consolidating them into fewer or what

Final checklist