integrate McLeish et al. + Lys et al.: reposition related work, rung-2 Muon note
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -198,16 +198,38 @@ checkpoint selection likewise failed to track generation accuracy.
|
||||
|
||||
## 4. Related work
|
||||
|
||||
Universal Transformers (adaptive depth); DEQ (fixed-point inference);
|
||||
Huginn / recurrent-depth latent reasoning (arXiv:2502.05171) — prelude/core/
|
||||
coda with input injection, trained from scratch; Mixture-of-Recursions
|
||||
(arXiv:2507.10524) — learned per-token depth; Relaxed Recursive Transformers
|
||||
(arXiv:2410.20672) — uptraining tied layers with per-loop LoRA; Coconut —
|
||||
latent chain-of-thought; pause tokens (Goyal et al.) — token-space silent
|
||||
compute. Distinct here: the recurrence is **retrofitted onto a frozen model
|
||||
at adapter cost**, its location is **chosen by an interpretability signal**,
|
||||
and the same signal **verifies** the added computation. Our pause-token and
|
||||
weights controls connect directly to that literature's baselines.
|
||||
Two recent papers bracket this work. **McLeish et al. (arXiv:2511.07384)**
|
||||
retrofit depth-recurrence into pretrained 1B models via layer surgery +
|
||||
continued pretraining (~50B tokens, all parameters, Muon, recurrence
|
||||
curriculum to r=32): the generic claims "retrofitted recurrence works and
|
||||
beats the non-recurrent parent" and "pretrain-then-convert" are theirs, at
|
||||
~5 orders of magnitude more training cost than ours. They name layer choice
|
||||
as an open problem; our lens-derived band with its causal backing (anchor
|
||||
cliff at L14, tap invariance, wrong-band ≈ 0, KV-sharing hazard) is a direct
|
||||
answer to it. Unlike their surgery (which needs a healing phase), our k=0
|
||||
exactly recovers the base model. **Lys et al. (arXiv:2602.14759)** loop
|
||||
frozen models training-free and show naive looping degrades (distribution
|
||||
shift) while interpolating with the un-looped state rescues it — independent
|
||||
convergent evidence for our anchor-dominant merge; their whole setting
|
||||
corresponds to the untrained cell of our attribution table (17.9% hard =
|
||||
our FF/untrained level), evaluated by likelihood rather than execution.
|
||||
|
||||
Earlier lineage: Universal Transformers (adaptive depth); DEQ (fixed-point
|
||||
inference); Huginn (arXiv:2502.05171) — prelude/core/coda from scratch;
|
||||
Mixture-of-Recursions (arXiv:2507.10524) — learned per-token depth; Relaxed
|
||||
Recursive Transformers (arXiv:2410.20672) — uptrained tied layers; Coconut —
|
||||
latent CoT; pause tokens (Goyal et al.) — token-space silent compute, whose
|
||||
trained-adapter variant proved a near-match for our loop on MBPP (§3.2).
|
||||
|
||||
What remains distinct here: **interpretability-derived loop placement with
|
||||
causal validation** (answering McLeish et al.'s open problem); **a 1.6M-param
|
||||
trained merge on a fully frozen base** (between Lys et al.'s free end and
|
||||
McLeish et al.'s full-retraining end, and the only one of the three where
|
||||
the base model is provably untouched); **prompt-only latent planning with
|
||||
bit-exact KV-cache write-in and zero decode cost**; **the attribution
|
||||
ladder** (untrained / weights / pause / loop / explicit plan) — neither
|
||||
bracket paper runs compute-matched token-space controls; and **difficulty-
|
||||
adaptive depth via the STaR-label gate**, named as future work in both.
|
||||
|
||||
## 5. Limitations
|
||||
|
||||
|
||||
Reference in New Issue
Block a user