From 7ac365271a33466a88c07e11834a1215cac0ba91 Mon Sep 17 00:00:00 2001 From: Nils Date: Tue, 14 Jul 2026 10:24:12 +0200 Subject: [PATCH] related_work: unify anchoring across McLeish/Lys/ours; link 12B alpha miscalibration to their stability results Co-Authored-By: Claude Fable 5 --- related_work/relevant_to_us.md | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/related_work/relevant_to_us.md b/related_work/relevant_to_us.md index dd251d4..88c495b 100644 --- a/related_work/relevant_to_us.md +++ b/related_work/relevant_to_us.md @@ -130,3 +130,31 @@ tiny trained merge on a frozen base + prompt-only latent planning with zero decode cost + a full attribution ladder + difficulty gating.** Neither paper touches any of the last three items; both strengthen the premise that band looping is a real mechanism rather than a curiosity. + +## One mechanism, three training regimes (anchoring) + +All three loop-entry designs are the same idea — **re-ground the looped +state in the baseline forward pass** — differing only in how much training +supports it: + +| regime | anchor form | training | outcome | +|---|---|---|---| +| Lys naive | none (α_state = 1) | zero | collapse (off-manifold drift) | +| Lys regularized | fixed inference-time interpolation η·h⁽⁰⁾ + (1−η)·h⁽ᵗ⁾ | zero | modest, consistent gains | +| **ours (Paper A)** | fixed mix (1−α)e + α·ŝ + trained zero-init MLP([e;ŝ]) | 1.6M params, frozen base | large gains at 2B-effective | +| McLeish | e concatenated every iteration through a **learned** adapter | all params, 50B tokens | gains at scale; anchor constraint absorbed by retraining | + +Reading: how much off-manifold state the frozen band tolerates is bounded, +and anchor strength must make up for whatever training does not provide. +Lys's naive-looping collapse is the zero-training limit of the phenomenon; +McLeish's full retraining is the opposite limit, where the manifold itself +moves to accommodate the loop and explicit anchoring dissolves into learned +injection. Our **12B α-miscalibration** (α=0.3 tuned at 2B; untrained loop +craters 72.6→23.0 at 12B) is then not a quirk but the same stability law +seen mid-spectrum: the tolerable loop share depends on the model, and a +coefficient tuned on one substrate overdrives another. This connects +directly to Lys's distribution-shift account and predicts the queued +α∈{0.1, 0.15} 12B arm should recover much of the loss; their softmax +auto-alignment (adaptive η, training-free) is the natural fallback if no +fixed α transfers. Worth a paragraph in Paper A (design justification + +12B analysis) and Paper B (stability mechanism).