3.7 KiB
Pre-registered protocol: unified adapter eval (written before any test numbers)
Date: 2026-07-13, after val@499, before step-799 completion. No test-set number for the unified adapter exists at time of writing.
-
Primary loop depth: k=2, for both tasks. Chosen on val CE with the tie-breaking rule: prefer the SMALLEST k whose hard-cell val CE is within 0.01 nats of the best k. (Current val: MBPP hard k2−k4 = 0.004, GSM hard k2−k4 = 0.006 → both ties → k=2.) The full k-curve is secondary/descriptive.
-
Checkpoint selection criterion (scalar, fixed now): mean of the two hard-cell val CEs at k=2, tasks weighted equally: crit = (gsm_hard_k2 + mbpp_hard_k2)/2. Lowest crit among saved checkpoints wins. (At writing: step 499, crit = (0.475+0.205)/2 = 0.340.)
-
Primary endpoints: (a) MBPP test pass@1 hard-bucket at k=2 vs k=0; (b) GSM8K test accuracy hard-bucket at k=2 vs k=0. McNemar, paired by item. Overall accuracy is secondary (known to be underpowered at n=250/256).
-
Same-harness rule: all k, INCLUDING k=0 baselines, measured by the prompt-only fast-path scripts (
generate_frozen_prompt; k=0 = plain cached generate inside the same function). No numbers carried over from the full-position-loop harness. -
Known missing control (not covered by this run): a same-size, no-recurrence adapter (h -> h + MLP(h) at the L13->L14 boundary, no loop, no band re-run) trained on identical data/objective. Until it exists, "the loop does the work (vs. 1.6M new weights anywhere doing it)" is NOT established. Queued as the next training run. Note the k=0 column is gated off by construction and is a sanity check only — it is not this control.
-
Symmetric interference check (missing): dedicated GSM8K prompt-only adapter as the reference for "unified costs GSM nothing". Queued. Until then the no-interference claim is one-directional (MBPP side only).
-
Band-location ablation (pre-registered 2026-07-13, before any arm ran). Arms, all else identical (adapter size/init, data, curriculum, k, scripts; MBPP): early L2-12, mid-narrow L17-27, late L24-34 (width-matched, 11 layers); shifted L6-22 (width-matched to the original 17). Reference: workspace L14-30 (already run, 3 seeds). Prediction: workspace-centered arms (L14-30, L17-27) exceed early/late on hard-bucket pass@1 at k=2-4 by a wide margin; shifted intermediate. Falsification: near-parity across arms demotes the lens claim from "locates where to loop" to "convenient discovery tool"; to be reported either way. Primary readout: hard-bucket pass@1 at k=4, e400 checkpoints throughout.
-
Language commitments for the writeup: the k0->k1 CE collapse (e.g. 4.36->0.18) is format/template learning expected from any trained adapter and must not be quoted as evidence of routing/planning; informative comparisons are within k>=1 cells only. Depth ordering k2 vs k4 deltas (0.001-0.006 nats) are inside checkpoint jitter and must be described as "k>=2 fits hard items equally well; k=1 slightly worse."
-
Anchor/entrance sweep (pre-registered 2026-07-14 ~03:00, before any arm ran). Arms: bands (13,30), (12,30), (11,30) — injection point shifted up from L14 at fixed tap L30; plus tap-23 = (14,23). All E2B/MBPP, same recipe, e400, primary readout hard-bucket pass@1 at k=4. Competing predictions: (a) "L14 special" (last full-attention KV-computing layer, lens boundary) → anchor-13 drops; (b) "KV-channel count" (anchors 11-13 add 1-3 extra KV-recomputing attention channels) → holds or improves. Body-length confound noted: earlier anchors lengthen the loop body; if results shift, run matched-length control (12,28) before interpreting.