Sibling Draft Review for PR #406

Date: 2026-04-12
Reviewer: Claude Agent (for Mato/@MatoTeziTanka)
NEEDS_LLM_REVIEW: Yes
Status: Ready for detailed code review

PR Summary

Title: Non-record: 11L XSA4 + EMA + SDTTT (3-seed mean val_bpb=1.1287)
State: OPEN
Mato blocking comment: NO

Marker Analysis

Architecture Changes

train_gpt.py modifications: ~1840 lines changed
Quantization: True
EMA: True
LoRA: False
GPTQ: True

Assessment

⚠ REQUIRES_REVIEW - Custom modifications detected:

• TTT (Test-Time Training) integration detected
• Custom tokenizer patterns found (3 instances)

Recommendation

REVIEW — Detailed code inspection needed before merge

Review Checklist

• Code style and formatting (PEP 8, consistency)
• Numerical stability of modifications
• Integration with existing pipeline
• Reproducibility (seed handling, RNG control)
• Performance metrics reported correctly
• No undocumented dependencies

Next Steps

1. Full train_gpt.py code review
2. Validate loss computation and gradients
3. Check for integration issues with existing pipeline
4. Verify metrics reproducibility

Generated for: Mato (@MatoTeziTanka)
Scope: NEEDS_LLM_REVIEW sweep
Action: Draft only—no posts to GitHub

Recommendation to @cocohearts @valerio-oai @0hq @yuzhougu-oai @notapplica: MERGE-pending standard record-track checks.

Reviewed by @MatoTeziTanka — The Agora. Classification via sibling-session agent (Haiku-backed). This review was drafted from a template and spot-checked before posting — if the template misread your code, please call it out so I can iterate the classifier.

Community Review — SDTTT + BigramHash + Depth Recurrence + Int6 QAT

Compliance flag: Pre-Quant TTT violation (SDTTT active)

The submitted score uses SDTTT (Self-Distillation TTT), confirmed active in the train log. SDTTT runs 2 epochs of SGD over all val_tokens before the scored sliding-window eval. Sequence from log: post_swa val_bpb:1.1448 → SDTTT 2-epoch adapt → post_sdttt val_bpb:1.1452 → final_int6_sliding_window val_bpb:1.1280 (submitted score). No BPB score is taken before adaptation completes — all val data is seen twice before the score is recorded. This violates score-first discipline.

Note on BigramHash: The BigramHash implementation xor(36313 * t[i], 27191 * t[i-1]) uses context tokens only (current + previous input), which is the legal pattern. No n-gram family bug.

Verdict: CLOSE — Pre-Quant TTT violation (SDTTT active, 2-epoch SGD on val_tokens before scoring).

Recommendation to @cocohearts @valerio-oai @0hq @yuzhougu-oai @notapplica: Recommend CLOSE unless the author resubmits with SDTTT disabled (the pre-SDTTT score of 1.1448 would be clean).

Reviewed by @MatoTeziTanka — The Agora. Compliance audit via LLM agent (Sonnet) reviewing full train_gpt.py source, with manual correction on BigramHash classification. If this review misread your code, please call it out so I can re-audit manually.

@MatoTeziTanka Thank you for the review. You are right, and the trace sequence you pulled from the train log is exactly what happened: SDTTT ran 2 epochs of SGD on val_tokens before the scored sliding window eval. Not disputing the flag.

Also appreciate you clearing the BigramHash implementation separately, that was a thoughtful distinction to make manually on top of the LLM audit.

I am working on the fix now. Two options I am weighing:

1. Push a commit to this branch that disables SDTTT by default and re-reports 1.1448 (the clean pre-SDTTT number from the same run log you quoted)
2. Close this PR and submit a new one with SDTTT rewritten to adapt on a held-out training slice instead of val_tokens, similar to the fix I pushed for PR Non-record: Universal Transformer + Adaptive Density (val_bpb 1.4390) #1193 / new submission Non-record: Universal Transformer + Legal Pre-Quant TTT (Training-Slice Variant) #1554

I will update this thread once the fix lands. Also doing a proactive audit of my other 5 research PRs (#1191, #1192, #1194, #1195, #1196, #1197) for the same pattern so I can self-flag anything else I find before it reaches your queue.

Thanks again for the careful review via The Agora. The feedback loop is working.

Fix pushed in commit 14bda5f.

Went with the in-place fix (Option 1 from my previous comment): updated README and submission.json to report the pre-SDTTT DIAGNOSTIC post_swa numbers from the same 3-seed runs. These are the clean predecessors of the tainted sliding_window numbers, computed on EMA-averaged weights before any SDTTT adaptation touched val_tokens.

Updated results

The train_gpt.py in the records folder actually already had sdttt_enabled defaulting to "0" (line 140), so running the committed script as-is produces legal results. The illegal numbers came from runs where SDTTT_ENABLED=1 was set as an env var override. No code change needed for legality, only the reported numbers needed fixing.

Audit of other research PRs

I also grepped my other 5 non-record research PRs (#1191, #1192, #1194, #1195, #1196, #1197) for the same pattern. They all use val_tokens only inside an eval_val() function under torch.inference_mode() with no .backward() calls. No TTT-on-val, no SDTTT, no hidden eval-time training. Those should be clean by the same standard.

Thanks again for the careful review.