A procedurally generated benchmark for compositional algebraic reasoning in language models.

ALGEBRAID generates an unlimited, non-memorisable supply of reasoning tasks built on five algebraic structures. Every answer is derived by algebraic construction; tasks that carry a solution trace (intra-structure, inter-structure, field arithmetic, adversarial, intermediate) are additionally re-derived by an independent proof engine. Tasks without traces (conceptual, rule induction) are verified by construction. Tasks are seed-deterministic: the same seed always produces the same task set.

The proof engine verifies algebraic trace consistency — it re-applies each named operation from first principles and checks every intermediate value. It does not re-read the prompt. A separate prompt-trace alignment check in the validator warns when a trace operation references an element that does not appear literally in the prompt (catches verbalizer bugs that the proof engine cannot detect).

pip install -e .

No provider SDK is bundled. Install only what you need:

pip install -e ".[openai]"      # OpenAI API and custom HTTP endpoints
pip install -e ".[anthropic]"   # Anthropic API

Set the corresponding API key before running:

export OPENAI_API_KEY=sk-...
export ANTHROPIC_API_KEY=sk-ant-...

# Generate a task set (validated automatically)
algebraid generate --seed 42
# -> ./data/algebraid_s42_20260225_e50604.jsonl

# Full pipeline: run model, evaluate, prove, analyze
algebraid pipeline ./data/algebraid_s42_20260225_e50604.jsonl \
    --model gpt-4.1-nano --adapter openai
# -> ./runs/20260225_143012_gpt_4_1_nano_algebraid_s42_.../

Full CLI reference: docs/cli.md

Seven generator families are available via --families (all seven are included by default):

adversarial and intermediate tasks are generated as intra-structure chains. They share the intra-structure composition family label in the task data and appear under that label in accuracy_by_family reports. They are distinguished from standard intra tasks by their dimension tag (adversarial or intermediate_state), which appears in accuracy_by_dimension reports.

Each task can be expressed through a narrative skin that maps abstract algebra to a real-world context. Skins are applied at generation time and affect prompt wording only, not the underlying computation.

Every task is tagged with one compositional dimension. The first four follow Hupkes et al. (2020); the remaining three are ALGEBRAID-specific. Evaluation reports break down accuracy by dimension via accuracy_by_dimension.

Every task carries three algebraic complexity scores computed independently of the model:

Two task-specific metrics are available as standalone functions:

run_analysis(report) returns a structured dict with five analyses plus metadata:

Standalone advanced functions (not in run_analysis output):

verify_set(task_set) returns a dict with the following keys:

from algebraid import (
    AlgebraidGenerator, AlgebraidEvaluator, EvalReport,
    TaskValidator, verify_set, run_analysis,
    split_by_depth, split_by_commutativity,
)

# Generate
# API defaults: depths=[1,2,3,4,5], tasks_per_depth=10, families=all seven
gen = AlgebraidGenerator(seed=42)
task_set = gen.generate(
    depths=[1, 2, 3, 4],
    tasks_per_depth=20,
    families=["intra", "inter", "conceptual", "adversarial"],
)

# Validate (includes prompt-trace alignment check)
val_report = TaskValidator().validate_set(task_set)
print(f"{val_report['passed']}/{val_report['total']} passed")

# Independently verify trace-based answers
proof = verify_set(task_set)
print(f"Trace-verified: {proof['trace_verified']}  "
      f"proof rate: {proof['proof_rate']}%")

# Evaluate (given a predictions dict)
predictions = {"AG-abc123": "5"}
evaluator = AlgebraidEvaluator()
report = evaluator.evaluate(task_set, predictions, model_name="my-model")
report.print_summary()

# Structured analysis: 5 analyses — depth, family, dimension, complexity,
# hallucination onset
analysis = run_analysis(report)

# Access the five structured analyses
print(analysis["accuracy_by_depth"]["curve"])          # per-depth accuracy
print(analysis["accuracy_by_family"])                  # all families
print(analysis["accuracy_by_dimension"])               # all dimensions
print(analysis["complexity_analysis"]["by_depth"])     # complexity metrics
print(analysis["hallucination_onset"]["note"])         # hallucination summary

# Optional: advanced standalone analyses (not in run_analysis output)
from algebraid.analysis import fit_scaling_law_by_family, find_phase_transition
by_family = fit_scaling_law_by_family(report)
for fam, fit in by_family.items():
    if fit["alpha"] is not None:
        print(f"{fam}: alpha={fit['alpha']:.2f}  R²={fit['r2']:.3f}")

# Generalisation splits
train, test = split_by_depth(task_set, train_max_depth=2)
train, test = split_by_commutativity(task_set)

# Load a saved report and re-analyze
with open("eval_report.json") as f:
    import json
    report = EvalReport.from_dict(json.load(f))
analysis = run_analysis(report)

src/algebraid/
    primitives/         Z_n, S_n, D_n, GF(p), Q_8, abstract base
    composers/          FunctionComposition, DirectProduct
    tasks/              Verbalization, answer verification, schema validation
    task_model.py       Task, TaskSet, TaskFamily, CompositionDimension
    generator.py        Procedural task generation (7 families)
    evaluator.py        AlgebraidEvaluator, EvalReport
    proof.py            Independent algebraic verifier
    analysis.py         Six structured analyses (depth, family, dimension, complexity, error taxonomy, hallucination onset); standalone: fit_scaling_law_by_family(), find_phase_transition()
    complexity.py       Algebraic complexity metrics
    splits.py           Train/test split strategies
    skins.py            12 semantic skins
    adapters.py         OpenAI, Anthropic, HuggingFace, custom HTTP adapters
    cli.py              8-command CLI
tests/
    test_primitives.py  Group axiom verification for all 5 structures
    test_composers.py   Composition and direct product tests
    test_generator.py   Determinism, schema, and grading tests
    test_complexity.py  Algebraic complexity metric tests
    test_task_model.py  Task/TaskSet serialization round-trip tests
    test_verbalizer.py  Prompt generation and skin integration tests
    test_verifier.py    Answer extraction and checking tests
    test_skins.py       Semantic skin coverage tests
    test_evaluator.py   EvalReport scoring and ceiling tests
    test_analysis.py    Error taxonomy, stability, accuracy by family, run_analysis key tests
    test_splits.py      Train/test split strategy tests

MIT. See LICENSE.