Research Expansion: Ablation Results

Ran an overnight ablation study on DGX Spark GB10 to expand on this submission. 200 training steps, sp1024, no torch.compile.

Results

Finding

H-Net is the fastest architecture tested at 513ms per step while still reaching 2.06 BPB (second best BPB overall after SSM). But the chunker configuration makes zero difference. All three variants produce identical BPB to 4 decimal places.

The chunker is learning the identity function regardless of what you do to it. Even with a boundary regularizer forcing chunk transitions, BPB does not change. This suggests the model routes around the chunking layer rather than using it.

Possible next steps: initialize the chunker with non-trivial boundaries (hard init from BPE token edges), or use hard chunking with straight-through estimators. Both untested here.

Full raw data and logs: https://gist.github.com/dentity007/324ac35505c27acd18e7ffb468f4fa08

Community Review — Non-record: H-Net Dynamic Chunking — Learned Tokenization Layer (val_bpb 1.3587)

Compliance: LOOKS CLEAN — pure-neural submission, no TTT/SLOT/n-gram-cache

Summary PR #1191 ("HNet_DynamicChunking_LearnedTokenization") adds a learned dynamic chunking layer (H-Net style) to the standard GPT baseline. The submission is architecturally clean with no illegal techniques detected. ## Checklist ### ILLEGAL N-gram family bug — NOT PRESENT No n-gram, bigram, hash, or XOR operations anywhere in the file. No target IDs XOR'd into any hash key. Search confirmed zero hits for ngram, bigram, hash, xor. ### ILLEGAL Pre-Quant TTT (multi-epoch on val_tokens without score-first) — NOT PRESENT val_tokens is loaded at line 891 and consumed exclusively by eval_val() (called at lines 1075 and 1197). eval_val() runs under torch.inference_mode() (line 261) with model.eval() (line 260) and model.train() restored at line 288. No .backward() call touches val_tokens. No multi-epoch loop over validation data exists. ### LEGAL Score-first TTT (PR #1413 pattern, is_last_chunk guard) — NOT PRESENT No TTT machinery of any kind. No is_last_chunk, no score-first guard, no adaptation loop. ### SCORED-REGION SLOT — NOT PRESENT No evidence of a held-out scored-region slot exploitation pattern. ### Architecture The novel component is DynamicChunker (lines 520–545) and DynamicChunkerStack (lines 548–557), inserted after tok_emb + rms_norm at lines 778–779 in GPT.forward(). It uses a learned boundary_proj (Linear dim*2 → 1) and chunk_mixer (Linear dim → dim) to soft-blend adjacent token embeddings. This is a purely architectural change — learned parameters trained on train data only — with no data leakage. ### Training loop Standard: train_loader feeds train shards (line 1011, 1109), val evaluation is read-only (lines 1072–1092), quantization roundtrip eval at lines 1197–1213. No...

Verdict: LOOKS CLEAN.

Recommendation to @cocohearts @valerio-oai @0hq @yuzhougu-oai @notapplica: MERGE pending the usual record-track checks (3-seed validation, under-16MB artifact cap, ≤600s train + ≤600s eval on 8×H100 SXM). No compliance flags from the audit — this looks like a clean pure-neural submission.

Reviewed by @MatoTeziTanka — The Agora. Compliance audit via LLM agent (Sonnet) reviewing full train_gpt.py source, cross-checked against deterministic AST classifier. If this review misread your code, please call it out so I can re-audit manually.

Thanks @MatoTeziTanka for the thorough audit. Especially appreciate you flagging the specific lines (891, 1075, 1197, 261, 288) for val_tokens flow and confirming the inference_mode boundary. That is exactly the kind of line-by-line verification that makes these reviews trustworthy.

One note on the architecture finding in case it's useful context for the merge decision: my own Spark ablation showed that the H-Net chunker learns the identity function regardless of configuration. All three variants (default, large chunker, boundary regularizer) produced identical BPB to 4 decimal places, which is documented in the PR body. The architecture is legal and the forward pass works, but the research takeaway is that the chunker as currently wired needs either hard chunking or stronger initial boundaries to produce non-trivial behavior. Worth noting for anyone who picks this up as a starting point.