integrate McLeish et al. + Lys et al.: reposition related work, rung-2 Muon note
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -255,6 +255,16 @@ passing code, CE on code tokens, curriculum k=1 easy / 2 mixed / 4 hard; the
|
||||
|
||||
### Rung-2 design note: depth-graded band unfreezing
|
||||
|
||||
**Optimizer note (from McLeish et al., arXiv:2511.07384):** use **Muon**, not
|
||||
AdamW, when unfreezing the band — they report AdamW loss-spikes to NaN when
|
||||
training through recurrence; adapter-only training didn't hit this, band-LoRA
|
||||
likely will. Also adopt their FLOPs convention for recurrent models
|
||||
(FLOPs = (6N₁+2N₂)D, N₁ = params with grads, N₂ = forward-only) in any
|
||||
compute-matched comparison. **Adaptive-α candidate (from Lys et al.,
|
||||
arXiv:2602.14759):** their training-free softmax auto-alignment interpolation
|
||||
is a cheap ablation against fixed α — directly relevant to the 12B α
|
||||
miscalibration.
|
||||
|
||||
When the band itself is unfrozen (per-iteration LoRA), unfreeze
|
||||
**entrance-faded**: full trainability at L14 decaying to frozen by ~L22
|
||||
(e.g. LoRA α × max(0, 1−(ℓ−14)/8)). Rationale: the only novel inputs in the
|
||||
|
||||
Reference in New Issue
Block a user