104 lines
6.6 KiB
Markdown
104 lines
6.6 KiB
Markdown
# Pre-registered protocol: unified adapter eval (written before any test numbers)
|
||
|
||
Date: 2026-07-13, after val@499, before step-799 completion. No test-set
|
||
number for the unified adapter exists at time of writing.
|
||
|
||
1. **Primary loop depth: k=2, for both tasks.** Chosen on val CE with the
|
||
tie-breaking rule: prefer the SMALLEST k whose hard-cell val CE is within
|
||
0.01 nats of the best k. (Current val: MBPP hard k2−k4 = 0.004, GSM hard
|
||
k2−k4 = 0.006 → both ties → k=2.) The full k-curve is secondary/descriptive.
|
||
2. **Checkpoint selection criterion (scalar, fixed now):** mean of the two
|
||
hard-cell val CEs at k=2, tasks weighted equally:
|
||
crit = (gsm_hard_k2 + mbpp_hard_k2)/2. Lowest crit among saved checkpoints
|
||
wins. (At writing: step 499, crit = (0.475+0.205)/2 = 0.340.)
|
||
3. **Primary endpoints:** (a) MBPP test pass@1 hard-bucket at k=2 vs k=0;
|
||
(b) GSM8K test accuracy hard-bucket at k=2 vs k=0. McNemar, paired by item.
|
||
Overall accuracy is secondary (known to be underpowered at n=250/256).
|
||
4. **Same-harness rule:** all k, INCLUDING k=0 baselines, measured by the
|
||
prompt-only fast-path scripts (`generate_frozen_prompt`; k=0 = plain
|
||
cached generate inside the same function). No numbers carried over from
|
||
the full-position-loop harness.
|
||
5. **Known missing control (not covered by this run):** a same-size,
|
||
no-recurrence adapter (h -> h + MLP(h) at the L13->L14 boundary, no loop,
|
||
no band re-run) trained on identical data/objective. Until it exists,
|
||
"the loop does the work (vs. 1.6M new weights anywhere doing it)" is NOT
|
||
established. Queued as the next training run. Note the k=0 column is
|
||
gated off by construction and is a sanity check only — it is not this
|
||
control.
|
||
6. **Symmetric interference check (missing):** dedicated GSM8K prompt-only
|
||
adapter as the reference for "unified costs GSM nothing". Queued. Until
|
||
then the no-interference claim is one-directional (MBPP side only).
|
||
7. **Band-location ablation (pre-registered 2026-07-13, before any arm ran).**
|
||
Arms, all else identical (adapter size/init, data, curriculum, k, scripts;
|
||
MBPP): early L2-12, mid-narrow L17-27, late L24-34 (width-matched, 11
|
||
layers); shifted L6-22 (width-matched to the original 17). Reference:
|
||
workspace L14-30 (already run, 3 seeds). Prediction: workspace-centered
|
||
arms (L14-30, L17-27) exceed early/late on hard-bucket pass@1 at k=2-4 by
|
||
a wide margin; shifted intermediate. Falsification: near-parity across
|
||
arms demotes the lens claim from "locates where to loop" to "convenient
|
||
discovery tool"; to be reported either way. Primary readout: hard-bucket
|
||
pass@1 at k=4, e400 checkpoints throughout.
|
||
8. **Language commitments for the writeup:** the k0->k1 CE collapse (e.g.
|
||
4.36->0.18) is format/template learning expected from any trained adapter
|
||
and must not be quoted as evidence of routing/planning; informative
|
||
comparisons are within k>=1 cells only. Depth ordering k2 vs k4 deltas
|
||
(0.001-0.006 nats) are inside checkpoint jitter and must be described as
|
||
"k>=2 fits hard items equally well; k=1 slightly worse."
|
||
|
||
9. **Anchor/entrance sweep (pre-registered 2026-07-14 ~03:00, before any arm
|
||
ran).** Arms: bands (13,30), (12,30), (11,30) — injection point shifted
|
||
up from L14 at fixed tap L30; plus tap-23 = (14,23). All E2B/MBPP, same
|
||
recipe, e400, primary readout hard-bucket pass@1 at k=4. Competing
|
||
predictions: (a) "L14 special" (last full-attention KV-computing layer,
|
||
lens boundary) → anchor-13 drops; (b) "KV-channel count" (anchors 11-13
|
||
add 1-3 extra KV-recomputing attention channels) → holds or improves.
|
||
Body-length confound noted: earlier anchors lengthen the loop body; if
|
||
results shift, run matched-length control (12,28) before interpreting.
|
||
|
||
10. **L9 discriminator arm (pre-registered 2026-07-14 ~10:00, before running).**
|
||
Band (9,30): anchor at L9 — the only other full-attention, KV-computing
|
||
layer below the boundary — deep in the lens's sensor regime, tap fixed at
|
||
L30. Separates the two cliff explanations: (a) "lens boundary" predicts
|
||
catastrophic (like anchors 11-13: 25-34% overall); (b) "full-attention
|
||
KV-layer entry" predicts partial recovery (clearly above the L11-13
|
||
trend, i.e. >40% overall or hard >25%). Registered prediction: (a) —
|
||
the sensor-region content dominates; layer type does not rescue it.
|
||
Same recipe/checkpoint/eval as the anchor sweep (250 items, ks 0,2,4).
|
||
|
||
---
|
||
|
||
# Outcomes vs pre-registrations (scored 2026-07-14, after all arms completed)
|
||
|
||
1. **k=2 primary depth** — held. All primary comparisons reported at k=2;
|
||
k-curves descriptive. k≥2 plateau confirmed (k=8 gen-eval flat).
|
||
2. **Checkpoint criterion** — applied as written for the unified adapter.
|
||
Separately reported: val-CE is a poor proxy for generation accuracy;
|
||
later arms therefore pre-committed to fixed steps (e400) instead.
|
||
3. **Primary endpoints (unified adapter, k=2 vs k=0)** — (a) MBPP hard
|
||
3.6% → 28.6% (direction as predicted); (b) GSM hard 0% → 6.3%, overall
|
||
10.5% → 9.0% (no overall win — the math boundary result). Both reported.
|
||
4. **Same-harness rule** — held throughout (all final tables fast-path,
|
||
k=0 included).
|
||
5. **Missing weights control** — run: trained FF adapter = 17.9% hard,
|
||
exactly the untrained-loop level. Loop-vs-weights gap established.
|
||
6. **Symmetric interference check** — run (dedicated GSM adapter);
|
||
mixed-task training regressed both tasks; reported as negative result.
|
||
7. **Band-location ablation** — prediction CONFIRMED with a caveat:
|
||
L14-30 hard 43.6% ≫ early L2-12 (23.6%, overall destroyed 22.8%) and
|
||
shifted L6-22 (21.8%, overall 29.0%). Caveat discovered: L17-27 and
|
||
L24-34 are structurally null (KV sharing; k>0 ≡ k=0 bit-identical), so
|
||
the "mid-narrow beats late" half of the prediction was untestable at
|
||
E2B; the 12B replication (no shared KV) carries that weight instead.
|
||
8. **Language commitments** — honored in PAPER.md (k0→k1 CE collapse not
|
||
cited as planning evidence; k2-vs-k4 nats described as jitter).
|
||
9. **Anchor/entrance sweep** — prediction (a) "L14 special" CONFIRMED:
|
||
anchor-13 hard 17.9%/overall 34.4%; 12: 28.6%/30.8%; 11: 25.0%/25.2%;
|
||
monotone collapse below the boundary. Tap-23 arm died in training
|
||
(never rerun); exits 27/30/32/34 within seed noise, so exit choice is
|
||
free. Matched-length control not needed (results did not shift with
|
||
body length in the informative direction).
|
||
10. **L9 discriminator** — registered prediction (a) CONFIRMED: band
|
||
(9,30) overall 14.0-21.4%, hard ≤21.4% — catastrophic, like anchors
|
||
11-13, despite L9 being a full-attention KV-computing layer. The lens
|
||
boundary, not layer type, gates the retrofit.
|