related_work: unify anchoring across McLeish/Lys/ours; link 12B alpha miscalibration to their stability results
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -130,3 +130,31 @@ tiny trained merge on a frozen base + prompt-only latent planning with zero
|
|||||||
decode cost + a full attribution ladder + difficulty gating.** Neither paper
|
decode cost + a full attribution ladder + difficulty gating.** Neither paper
|
||||||
touches any of the last three items; both strengthen the premise that band
|
touches any of the last three items; both strengthen the premise that band
|
||||||
looping is a real mechanism rather than a curiosity.
|
looping is a real mechanism rather than a curiosity.
|
||||||
|
|
||||||
|
## One mechanism, three training regimes (anchoring)
|
||||||
|
|
||||||
|
All three loop-entry designs are the same idea — **re-ground the looped
|
||||||
|
state in the baseline forward pass** — differing only in how much training
|
||||||
|
supports it:
|
||||||
|
|
||||||
|
| regime | anchor form | training | outcome |
|
||||||
|
|---|---|---|---|
|
||||||
|
| Lys naive | none (α_state = 1) | zero | collapse (off-manifold drift) |
|
||||||
|
| Lys regularized | fixed inference-time interpolation η·h⁽⁰⁾ + (1−η)·h⁽ᵗ⁾ | zero | modest, consistent gains |
|
||||||
|
| **ours (Paper A)** | fixed mix (1−α)e + α·ŝ + trained zero-init MLP([e;ŝ]) | 1.6M params, frozen base | large gains at 2B-effective |
|
||||||
|
| McLeish | e concatenated every iteration through a **learned** adapter | all params, 50B tokens | gains at scale; anchor constraint absorbed by retraining |
|
||||||
|
|
||||||
|
Reading: how much off-manifold state the frozen band tolerates is bounded,
|
||||||
|
and anchor strength must make up for whatever training does not provide.
|
||||||
|
Lys's naive-looping collapse is the zero-training limit of the phenomenon;
|
||||||
|
McLeish's full retraining is the opposite limit, where the manifold itself
|
||||||
|
moves to accommodate the loop and explicit anchoring dissolves into learned
|
||||||
|
injection. Our **12B α-miscalibration** (α=0.3 tuned at 2B; untrained loop
|
||||||
|
craters 72.6→23.0 at 12B) is then not a quirk but the same stability law
|
||||||
|
seen mid-spectrum: the tolerable loop share depends on the model, and a
|
||||||
|
coefficient tuned on one substrate overdrives another. This connects
|
||||||
|
directly to Lys's distribution-shift account and predicts the queued
|
||||||
|
α∈{0.1, 0.15} 12B arm should recover much of the loss; their softmax
|
||||||
|
auto-alignment (adaptive η, training-free) is the natural fallback if no
|
||||||
|
fixed α transfers. Worth a paragraph in Paper A (design justification +
|
||||||
|
12B analysis) and Paper B (stability mechanism).
|
||||||
|
|||||||
Reference in New Issue
Block a user