RecurrentAdapter arm: Huginn-regime retrofit (learned A/B, noise h0, randomized depth) + pre-registration item 11

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Nils
2026-07-15 00:49:44 +02:00
co-authored by Claude Fable 5
parent 0fd93cb328
commit d2da8044c3
4 changed files with 94 additions and 9 deletions
+16
View File
@@ -101,3 +101,19 @@ number for the unified adapter exists at time of writing.
(9,30) overall 14.0-21.4%, hard ≤21.4% — catastrophic, like anchors
11-13, despite L9 being a full-attention KV-computing layer. The lens
boundary, not layer type, gates the retrofit.
11. **Recurrent-regime arm (pre-registered 2026-07-15, before training).**
Huginn-style retrofit on the frozen E2B band: RecurrentAdapter
(learned A,B init α·I/(1−α)·I + zero-init MLP), h0 = norm-scaled
noise, log-uniform random depth k∈[1,16], bptt=4, same data/steps/
checkpoint rule (e400 primary) as all merge arms. Eval ks 0,2,4,8,16,32
on the 250-item MBPP set. Competing predictions: (a) "amortization is
intrinsic to frozen-band retrofits" → performance plateaus by k≈4 at
or below the merge arm's level, no depth-monotone gain; (b) "fixed-
point behavior was an artifact of our fixed-shallow-k training"
(Huginn regime transfers) → monotone hard-bucket improvement past k=8
and reduced noise-seed sensitivity after training. Secondary readout:
path independence (two noise seeds → output agreement rate) at e400.
Known risk, stated in advance: 600 steps may be far too little for
this regime (McLeish et al. use ~50B tokens); a null here bounds the
cheap-retrofit budget only, not the regime.