unification paragraph: anchor coefficient as the stability dial across Lys/ours/McLeish
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -214,6 +214,20 @@ convergent evidence for our anchor-dominant merge; their whole setting
|
||||
corresponds to the untrained cell of our attribution table (17.9% hard =
|
||||
our FF/untrained level), evaluated by likelihood rather than execution.
|
||||
|
||||
**One mechanism, three regimes.** All three works are variants of a single
|
||||
design: mix the fed-back state with an anchor derived from the un-looped
|
||||
computation. Lys et al.'s inference-time moving average η·h⁽⁰⁾+(1−η)·h⁽ᵗ⁾ is
|
||||
an untrained anchor-mixing coefficient; our (1−α)e + α·ŝ + MLP([e;ŝ]) is its
|
||||
trained analogue (fixed mix + learned correction); McLeish et al.'s
|
||||
concatenated input injection is the fully learned limit, trained end-to-end.
|
||||
The anchor coefficient is the stability dial of frozen-band looping: Lys
|
||||
et al.'s naive-looping collapse is the zero-anchor (α→1) limit, their
|
||||
regularization gains are the untrained anchored regime, and our 12B failure
|
||||
at the 2B-tuned α=0.3 — with the untrained-substrate arm showing the damage
|
||||
is pre-training-of-the-adapter — is the same dial mis-set at a new scale.
|
||||
Stability of retrofitted recurrence appears to be governed by how strongly
|
||||
the loop is anchored, across all three training budgets.
|
||||
|
||||
Earlier lineage: Universal Transformers (adaptive depth); DEQ (fixed-point
|
||||
inference); Huginn (arXiv:2502.05171) — prelude/core/coda from scratch;
|
||||
Mixture-of-Recursions (arXiv:2507.10524) — learned per-token depth; Relaxed
|
||||
|
||||
Reference in New Issue
Block a user