related_work: unify anchoring across McLeish/Lys/ours; link 12B alpha miscalibration to their stability results

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Nils
2026-07-14 10:24:12 +02:00
co-authored by Claude Fable 5
parent 89df135631
commit 7ac365271a
+28
View File
@@ -130,3 +130,31 @@ tiny trained merge on a frozen base + prompt-only latent planning with zero
decode cost + a full attribution ladder + difficulty gating.** Neither paper decode cost + a full attribution ladder + difficulty gating.** Neither paper
touches any of the last three items; both strengthen the premise that band touches any of the last three items; both strengthen the premise that band
looping is a real mechanism rather than a curiosity. looping is a real mechanism rather than a curiosity.
## One mechanism, three training regimes (anchoring)
All three loop-entry designs are the same idea — **re-ground the looped
state in the baseline forward pass** — differing only in how much training
supports it:
| regime | anchor form | training | outcome |
|---|---|---|---|
| Lys naive | none (α_state = 1) | zero | collapse (off-manifold drift) |
| Lys regularized | fixed inference-time interpolation η·h⁽⁰⁾ + (1−η)·h⁽ᵗ⁾ | zero | modest, consistent gains |
| **ours (Paper A)** | fixed mix (1−α)e + α·ŝ + trained zero-init MLP([e;ŝ]) | 1.6M params, frozen base | large gains at 2B-effective |
| McLeish | e concatenated every iteration through a **learned** adapter | all params, 50B tokens | gains at scale; anchor constraint absorbed by retraining |
Reading: how much off-manifold state the frozen band tolerates is bounded,
and anchor strength must make up for whatever training does not provide.
Lys's naive-looping collapse is the zero-training limit of the phenomenon;
McLeish's full retraining is the opposite limit, where the manifold itself
moves to accommodate the loop and explicit anchoring dissolves into learned
injection. Our **12B α-miscalibration** (α=0.3 tuned at 2B; untrained loop
craters 72.6→23.0 at 12B) is then not a quirk but the same stability law
seen mid-spectrum: the tolerable loop share depends on the model, and a
coefficient tuned on one substrate overdrives another. This connects
directly to Lys's distribution-shift account and predicts the queued
α∈{0.1, 0.15} 12B arm should recover much of the loss; their softmax
auto-alignment (adaptive η, training-free) is the natural fallback if no
fixed α transfers. Worth a paragraph in Paper A (design justification +
12B analysis) and Paper B (stability mechanism).