integrate McLeish et al. + Lys et al.: reposition related work, rung-2 Muon note

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Nils
2026-07-14 10:24:17 +02:00
co-authored by Claude Fable 5
parent 7ac365271a
commit 8e7bd2019e
2 changed files with 42 additions and 10 deletions
+10
View File
@@ -255,6 +255,16 @@ passing code, CE on code tokens, curriculum k=1 easy / 2 mixed / 4 hard; the
### Rung-2 design note: depth-graded band unfreezing
**Optimizer note (from McLeish et al., arXiv:2511.07384):** use **Muon**, not
AdamW, when unfreezing the band — they report AdamW loss-spikes to NaN when
training through recurrence; adapter-only training didn't hit this, band-LoRA
likely will. Also adopt their FLOPs convention for recurrent models
(FLOPs = (6N₁+2N₂)D, N₁ = params with grads, N₂ = forward-only) in any
compute-matched comparison. **Adaptive-α candidate (from Lys et al.,
arXiv:2602.14759):** their training-free softmax auto-alignment interpolation
is a cheap ablation against fixed α — directly relevant to the 12B α
miscalibration.
When the band itself is unfrozen (per-iteration LoRA), unfreeze
**entrance-faded**: full trainability at L14 decaying to frozen by ~L22
(e.g. LoRA α × max(0, 1(ℓ−14)/8)). Rationale: the only novel inputs in the