RecurrentAdapter arm: Huginn-regime retrofit (learned A/B, noise h0, randomized depth) + pre-registration item 11
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -101,3 +101,19 @@ number for the unified adapter exists at time of writing.
|
||||
(9,30) overall 14.0-21.4%, hard ≤21.4% — catastrophic, like anchors
|
||||
11-13, despite L9 being a full-attention KV-computing layer. The lens
|
||||
boundary, not layer type, gates the retrofit.
|
||||
|
||||
11. **Recurrent-regime arm (pre-registered 2026-07-15, before training).**
|
||||
Huginn-style retrofit on the frozen E2B band: RecurrentAdapter
|
||||
(learned A,B init α·I/(1−α)·I + zero-init MLP), h0 = norm-scaled
|
||||
noise, log-uniform random depth k∈[1,16], bptt=4, same data/steps/
|
||||
checkpoint rule (e400 primary) as all merge arms. Eval ks 0,2,4,8,16,32
|
||||
on the 250-item MBPP set. Competing predictions: (a) "amortization is
|
||||
intrinsic to frozen-band retrofits" → performance plateaus by k≈4 at
|
||||
or below the merge arm's level, no depth-monotone gain; (b) "fixed-
|
||||
point behavior was an artifact of our fixed-shallow-k training"
|
||||
(Huginn regime transfers) → monotone hard-bucket improvement past k=8
|
||||
and reduced noise-seed sensitivity after training. Secondary readout:
|
||||
path independence (two noise seeds → output agreement rate) at e400.
|
||||
Known risk, stated in advance: 600 steps may be far too little for
|
||||
this regime (McLeish et al. use ~50B tokens); a null here bounds the
|
||||
cheap-retrofit budget only, not the regime.
|
||||
|
||||
Reference in New Issue
Block a user