Platform Evolution Engine — Distill the best from the combinatorial mess.

Retort applies statistical Design of Experiments (DoE) to systematically evaluate AI-assisted development tooling stacks. It generates fractional factorial designs across languages, coding agents, and frameworks, executes experiments in isolated playpens, scores the results, and promotes or retires stacks based on measured confidence.

Retort is pre-1.0 and under active development. The CLI surface, scoring metrics, and storage schema may change between commits. The code is published so others can read it, fork it, and reproduce experiments — not yet as a stable tool to depend on.

What works today: LocalRunner, all 8 built-in scorers (code_quality, test_coverage, test_quality, defect_rate, maintainability, token_efficiency, idiomatic, findings), fractional-factorial design generation, ANOVA + effects reporting, SQLite storage, resumable runs, parallel bulk evaluation (retort evaluate --workers N), and the evaluate-run skill pipeline that scores generated code against task requirements and produces per-run evaluation.md, findings.jsonl, and assessment.json.

What does not yet work end-to-end: DockerRunner (skeleton only — LocalRunner is the supported path), agents other than claude-code, the intake/scheduler paths. See Implementation Status for details.

Scoring gate: A run where tests don't execute scores 0 across all metrics — a Starlette-incompatible Python run that writes perfect code still fails if pytest can't import. test_coverage == 0 vetoes the entire ScoreVector. The findings scorer reads assessment.json produced by the evaluate-run + file-run-issues skill pipeline and applies a weighted penalty for critical/high/medium/low findings.

📊 Browse the live web report → (sortable leaderboard with per-stack drill-downs, token/cost data, and links to per-run code reviews)

Full data is also in experiment-1/reports/ — ANOVA, per-stack maturity, full CSV, and the same static-HTML web report. Below is the headline.

Setup: 6 languages (python, typescript, go, rust, java, clojure) × 2 models (opus, sonnet) × 2 tooling (none, beads) × 3 replicates = 72 runs against the bundled rest-api-crud task. Java + clojure added in a follow-up extension run. Final tally: 67 of 73 runs completed, 6 failed. Total cost ≈ $25, ≈ 25.8M tokens.

Evaluation scores (from retort evaluate + evaluate-run skill, April 2026): PenScore = 1.0 minus weighted findings penalty (critical×0.25, high×0.10, medium×0.03, low×0.01); ReqCov = fraction of TASK.md requirements implemented. Runs where tests didn't execute score 0.

Multiplicative ANOVA (default: log10 transform, since cost/tokens/duration scale by ratios not constants):

Switching from additive to multiplicative ANOVA surfaced model + tooling effects on the cost-like metrics that the additive model treated as noise. See reports/anova.txt.

Top stacks by maturity: Java sweeps quality 1.000 in all four model/tooling combinations. Go's sonnet/beads ties at quality 1.000 too. Generate the full maturity report with retort maturity --db experiment-1/retort.db.

Sortable + drill-downable in the web report. PenScore and ReqCov are from the April 2026 evaluate-run bulk evaluation. Bold = perfect penalty score.

Headlines:

• Java, Go, Rust, and Clojure/opus consistently hit PenScore 1.000 on this task — no findings above threshold.
• Python is the outlier. python/opus/none and python/sonnet/beads scored 0.50 and 0.40 due to a Starlette 1.0 compatibility break that prevented tests from executing (which zeroes all scores under the new test-gate rule). Other Python cells pass cleanly.
• Requirement coverage diverges from penalty score. go/sonnet/beads has PenScore 0.995 but ReqCov 0.500 — the code is high quality but only implements half the spec. The old code_quality scorer missed this.
• Beads helps only for Go. Pattern consistent with experiment-1 ANOVA findings.

📊 Web report →

A second experiment run against brazil-bench/benchmark-template — a much harder task: MCP server, CSV ingest of Kaggle data, BDD tests with 16 canonical requirements. 24 cells, 1 replicate each, screening pass. Results combine with experiment-1 to give cross-task ANOVA insights.

Single-task ANOVA on code_quality: only language significant (consistent with experiment-1).

Cross-task ANOVA (89 rows = experiment-1's 67 + experiment-2's 22, task as a factor):

The model:task interaction is the headline finding. Opus vs sonnet behaves differently on hard (brazil-bench) vs easy (rest-api-crud) tasks. The simple "best stack on rest-api-crud is best everywhere" assumption from experiment-1 doesn't fully generalize for the resource-cost dimensions.

PenScore and ReqCov from evaluate-run bulk evaluation. Brazil-bench is a harder task (MCP server + CSV ingest + BDD tests) so scores spread more widely.

Headlines for brazil-bench:

• Python sweeps clean — all four cells hit PenScore 1.000, reversing the experiment-1 pattern (Starlette issue is task-specific, not language-specific).
• Java/sonnet/none catastrophically fails (11 criticals, PenScore 0.000) — a single cell failure, not a model trend; the same model/tooling is fine in all other combinations.
• Rust struggles on the harder task — all four cells below 1.000, consistent with Rust's complexity wall on multi-component tasks.
• model:task interaction confirmed by evaluation scores: opus outperforms sonnet on brazil-bench for Go and Java; sonnet leads on rust/none.

Pareto frontier across both tasks (retort report pareto --data combined.csv --metric code_quality --metric -_cost_usd):

Every other stack is dominated. go / sonnet / beads is the only stack that no other stack beats on both quality AND cost simultaneously.

For prediction of unmeasured cells (when running fewer cells of a fractional design), use retort analyze --predict — emits 95% CIs for cells you didn't run, fitted from cells you did.

pip install only fetches the Python deps. To actually run experiments you also need:

git clone https://github.com/adrianco/retort.git
cd retort
pip install -e ".[dev,test]"

.devcontainer/ provisions Python 3.12 + the C++ toolchain + Node + Go + Rust + the claude and bd CLIs via post-create.sh. Open the repo in GitHub Codespaces or VS Code "Dev Containers: Reopen in Container" and the prereqs are handled automatically. You'll still need to authenticate claude interactively the first time.

retort --help                # CLI loads → Python deps OK
claude --version             # Claude CLI present
bd --version                 # Beads present (only needed for tooling=beads)

# Initialize a workspace
retort init my-eval
cd my-eval

# Edit workspace.yaml to define your factors, responses, and tasks
# Then generate a screening design matrix
retort design generate --phase screening --config workspace.yaml -o design.csv

# Execute experiment runs
retort run --phase screening --config workspace.yaml

# Compute main effects and interactions
retort report effects --db retort.db --matrix-id 1 --metric code_quality

# Evaluate a promotion gate
retort promote my-stack --from screening --to trial \
    --evidence '{"p_value": 0.05}' --config workspace.yaml

Retort runs standalone — pip install + claude CLI is enough to drive every command above. There is no Gas Town dependency.

That said, real experiments are long-running, parallelizable, and benefit from an orchestrator. Gas Town is the orchestrator we use during development; it adds:

• Parallel execution. gt sling dispatches a slice of the design to a polecat (a worker agent in its own git worktree). Multiple polecats share one retort.db via retort run --shard N/M --resume. The --shard partition is a deterministic hash, so two polecats never both pick the same cell, and per-run sqlite commits keep concurrent writers safe.
• Patrol + escalation. witness watches the merge queue; refinery patrols it; mail/escalations route to the mayor agent if anything sticks.
• Auto-evaluation. With gt + bd (beads) installed, the evaluate-run and file-run-issues skills file findings as tracked beads in your project — survives session resets and shows up in queries.

Pattern:

gt sling re-ucc retort --crew alpha   --args "retort run --phase screening --config experiment-2/workspace.yaml --resume --shard 0/4"
gt sling re-ucc retort --crew bravo   --args "retort run --phase screening --config experiment-2/workspace.yaml --resume --shard 1/4"
gt sling re-ucc retort --crew charlie --args "retort run --phase screening --config experiment-2/workspace.yaml --resume --shard 2/4"
gt sling re-ucc retort --crew delta   --args "retort run --phase screening --config experiment-2/workspace.yaml --resume --shard 3/4"

Without Gas Town, the same parallelism works in plain bash:

for s in 0 1 2 3; do
    nohup retort run --phase screening --config experiment-2/workspace.yaml \
        --resume --shard $s/4 > shard-$s.log 2>&1 &
done
wait

Caveats if you choose the gt path:

• Some current gt 0.12.0 rough edges (project-id mismatches, missing bd agent subcommand, gt dolt fix-metadata vs rig-config-sync disagreement) are documented in re-o5b in the beads tracker. None block work; they produce noisy telemetry warnings.
• Concurrent runs multiply per-second token usage. Keep the shard count to a value the Anthropic API tier comfortably supports — start at 2× and monitor rate-limit headers before going higher.

Retort workspaces are configured via workspace.yaml:

factors:
  language:
    levels: [python, typescript, go]
  agent:
    levels: [claude-code, cursor, copilot]
  framework:
    levels: [fastapi, nextjs, stdlib]

responses:
  - code_quality
  - token_efficiency
  - build_time
  - test_coverage

tasks:
  - source: bundled://rest-api-crud

playpen:
  runner: local            # 'local' is the supported path; 'docker' is a skeleton
  replicates: 3
  timeout_minutes: 30

design:
  screening_resolution: 3
  significance_threshold: 0.10

promotion:
  screening_to_trial: { p_value: 0.10 }
  trial_to_production: { posterior_confidence: 0.80 }

src/retort/
├── cli.py              # Click-based CLI entry point
├── analysis/           # ANOVA, Bayesian updating (conjugate NIG), Pareto frontier, residual diagnostics
├── config/             # Pydantic config schema and YAML loader
├── design/             # Factor registry and fractional factorial generator (pyDOE3)
├── playpen/            # Isolated execution: DockerRunner, task loading, prompt building
├── plugins.py          # Pluggy-based plugin system for custom scorers and runners
├── scoring/            # Score collection with pluggable scorers (code_quality, token_efficiency, build_time)
├── promotion/          # Lifecycle state machine, configurable gates, immutable changelog
├── reporting/          # Effects computation, export (text, JSON, CSV), and status dashboard
├── scheduler/          # Candidate intake, D-optimal augmentation, budget tracking, run queue
└── storage/            # SQLAlchemy models and Alembic migrations (SQLite)

• Factors: Variables under test (language, agent, framework) with discrete levels
• Design Matrix: Fractional factorial design that efficiently covers the factor space
• Playpen: Isolated Docker environment where each experiment run executes
• Scoring: Pluggable metrics collected from run artifacts
• Promotion: Evidence-based lifecycle transitions (candidate → screening → trial → production → retired)

Honest accounting. "Code exists" ≠ "tested end-to-end against real data."

• DockerRunner not validated — use runner: local in workspace.yaml for now
• Only claude-code is wired up as an agent
• idiomatic scorer makes a Claude haiku call per run (~$0.001/run); cached per workspace in .idiomatic_cache.json
• retort intake / D-optimal augmentation untested against a real candidate
• Bundled tasks cli-data-pipeline and react-dashboard exist but haven't been run end-to-end
• evaluate-run skill timeout is 600s per run (evaluate + file-run-issues chained in one call); runs on very large codebases may hit the limit
• 8 of 61 experiment-1 runs and 2 of 24 experiment-2 runs missed evaluation (rc=143 SIGTERM under load from concurrent haiku workers) — re-run with retort evaluate --workers 2 on the missing runs

# Install dev dependencies
pip install -e ".[dev,test]"

# Run tests
pytest

# Lint
ruff check src/ tests/

# Type check
mypy src/retort/

Apache-2.0 — see LICENSE.