From 6c39c38eb9d8ce96a5d29a13ccda2daf4f84c3d6 Mon Sep 17 00:00:00 2001 From: Nils Date: Tue, 14 Jul 2026 02:54:52 +0200 Subject: [PATCH] track protocol pre-registrations and interim report in git Co-Authored-By: Claude Fable 5 --- results-loop/PROTOCOL_UNIFIED.md | 56 +++++++++++++++++++ results-loop/REPORT_wave3.md | 95 ++++++++++++++++++++++++++++++++ 2 files changed, 151 insertions(+) create mode 100644 results-loop/PROTOCOL_UNIFIED.md create mode 100644 results-loop/REPORT_wave3.md diff --git a/results-loop/PROTOCOL_UNIFIED.md b/results-loop/PROTOCOL_UNIFIED.md new file mode 100644 index 0000000..7ea28e4 --- /dev/null +++ b/results-loop/PROTOCOL_UNIFIED.md @@ -0,0 +1,56 @@ +# Pre-registered protocol: unified adapter eval (written before any test numbers) + +Date: 2026-07-13, after val@499, before step-799 completion. No test-set +number for the unified adapter exists at time of writing. + +1. **Primary loop depth: k=2, for both tasks.** Chosen on val CE with the + tie-breaking rule: prefer the SMALLEST k whose hard-cell val CE is within + 0.01 nats of the best k. (Current val: MBPP hard k2−k4 = 0.004, GSM hard + k2−k4 = 0.006 → both ties → k=2.) The full k-curve is secondary/descriptive. +2. **Checkpoint selection criterion (scalar, fixed now):** mean of the two + hard-cell val CEs at k=2, tasks weighted equally: + crit = (gsm_hard_k2 + mbpp_hard_k2)/2. Lowest crit among saved checkpoints + wins. (At writing: step 499, crit = (0.475+0.205)/2 = 0.340.) +3. **Primary endpoints:** (a) MBPP test pass@1 hard-bucket at k=2 vs k=0; + (b) GSM8K test accuracy hard-bucket at k=2 vs k=0. McNemar, paired by item. + Overall accuracy is secondary (known to be underpowered at n=250/256). +4. **Same-harness rule:** all k, INCLUDING k=0 baselines, measured by the + prompt-only fast-path scripts (`generate_frozen_prompt`; k=0 = plain + cached generate inside the same function). No numbers carried over from + the full-position-loop harness. +5. **Known missing control (not covered by this run):** a same-size, + no-recurrence adapter (h -> h + MLP(h) at the L13->L14 boundary, no loop, + no band re-run) trained on identical data/objective. Until it exists, + "the loop does the work (vs. 1.6M new weights anywhere doing it)" is NOT + established. Queued as the next training run. Note the k=0 column is + gated off by construction and is a sanity check only — it is not this + control. +6. **Symmetric interference check (missing):** dedicated GSM8K prompt-only + adapter as the reference for "unified costs GSM nothing". Queued. Until + then the no-interference claim is one-directional (MBPP side only). +7. **Band-location ablation (pre-registered 2026-07-13, before any arm ran).** + Arms, all else identical (adapter size/init, data, curriculum, k, scripts; + MBPP): early L2-12, mid-narrow L17-27, late L24-34 (width-matched, 11 + layers); shifted L6-22 (width-matched to the original 17). Reference: + workspace L14-30 (already run, 3 seeds). Prediction: workspace-centered + arms (L14-30, L17-27) exceed early/late on hard-bucket pass@1 at k=2-4 by + a wide margin; shifted intermediate. Falsification: near-parity across + arms demotes the lens claim from "locates where to loop" to "convenient + discovery tool"; to be reported either way. Primary readout: hard-bucket + pass@1 at k=4, e400 checkpoints throughout. +8. **Language commitments for the writeup:** the k0->k1 CE collapse (e.g. + 4.36->0.18) is format/template learning expected from any trained adapter + and must not be quoted as evidence of routing/planning; informative + comparisons are within k>=1 cells only. Depth ordering k2 vs k4 deltas + (0.001-0.006 nats) are inside checkpoint jitter and must be described as + "k>=2 fits hard items equally well; k=1 slightly worse." + +9. **Anchor/entrance sweep (pre-registered 2026-07-14 ~03:00, before any arm + ran).** Arms: bands (13,30), (12,30), (11,30) — injection point shifted + up from L14 at fixed tap L30; plus tap-23 = (14,23). All E2B/MBPP, same + recipe, e400, primary readout hard-bucket pass@1 at k=4. Competing + predictions: (a) "L14 special" (last full-attention KV-computing layer, + lens boundary) → anchor-13 drops; (b) "KV-channel count" (anchors 11-13 + add 1-3 extra KV-recomputing attention channels) → holds or improves. + Body-length confound noted: earlier anchors lengthen the loop body; if + results shift, run matched-length control (12,28) before interpreting. diff --git a/results-loop/REPORT_wave3.md b/results-loop/REPORT_wave3.md new file mode 100644 index 0000000..cb264f3 --- /dev/null +++ b/results-loop/REPORT_wave3.md @@ -0,0 +1,95 @@ +# Interim report: workspace-loop retrofit, through wave 3 + +2026-07-13/14 · gemma-4-E2B-it (frozen throughout) · DGX Spark +All eval numbers: held-out test splits, greedy decode, execution/answer-verified. +Full protocol: `PROTOCOL_UNIFIED.md`; methods: `../WORKSPACE_LOOPING.md`. + +## Headline finding + +A 1.6M-parameter merge adapter (0.03% of the model) at the L13→L14 boundary, +trained ~30 min on ~280 self-labeled items, lets the frozen model **loop its +workspace band (L14–30) over the prompt k times before writing code** — +converting plan-dependent failures into successes: + +**MBPP full test set (500 items), hard bucket (55 plan-only items), k=4:** + +| seed | hard pass@1 | overall | +|---|---|---| +| baseline (k=0) | 5.5% | 51.8% | +| seed 0 | **43.6%** | 53.6% | +| seed 1 | **41.8%** | 53.8% | +| seed 2 | **30.9%** | 51.8% | + +Mean hard-bucket effect ≈ 7× baseline; overall at-or-above baseline for all +seeds (no net tax). Convergent mechanism evidence: the trained loop is a hard +fixed-point iteration (cos→1.000 by k≈4), and the J-lens shows latent-concept +sharpening across iterations (~8× over untrained control on the probe task). + +## Attribution: the loop is the ingredient (code), not the weights + +Same data, same parameter count, same boundary — only the mechanism varies +(MBPP 250-item subset, hard bucket): + +| arm | hard pass@1 | +|---|---| +| baseline | 3.6% | +| untrained loop (α-merge only) | 17.9% | +| trained NO-loop adapter (weights control) | 17.9% | +| trained loop | **42.9–46.4%** | + +The trained feedforward control lands exactly on the untrained-loop number: +~18% is what perturbation + format alignment buys; the remaining ~28 points +require the recurrence. + +## The boundary: math inverts the picture + +No recurrent variant beats plain weights on GSM8K. Four-arm grid (hard bucket): + +| GSM8K hard | prompt-side only | touches generation | +|---|---|---| +| feedforward weights | **11.8%** (best) | 4.7% (pause control) | +| recurrence | 6.3–8.7% (prompt-loop) | 9.4% (carry, beats FF 2× in-harness) | + +Two orthogonal effects: (1) perturbing free-running generation positions is +costly for either mechanism; (2) recurrence beats weights only where a state +must evolve (generation side), and loses where it doesn't (static prompt). +Interpretation: the loop performs **plan refinement**. Code needs a plan +(loop wins); math needs answer-time computation that one frozen band pass per +step cannot supply at 2B (CoT tokens remain load-bearing). Overall GSM +accuracy: no variant beats the 10.5% baseline. (Hard-bucket cells carry an +outcome-selection caveat — greedy labels; sampled relabeling in progress — +so the math story rests on the overall numbers.) + +## Secondary findings + +- **Unified (mixed-task) training regressed both tasks** vs dedicated + adapters (MBPP 46.8% vs 52.0% at pre-registered k=2; GSM no better). + CE-level parity between arms did NOT predict generation parity. +- **Gated deployment** (logistic probe on the k=0 workspace state, STaR + labels as supervision): 52.0% overall with easy items fully preserved + (97.5%); gate recall 64%, precision 19% — the current ceiling; threshold/ + feature work is CPU-only follow-up. +- **Frozen-prompt KV-cache equivalence**: looped prompt states are constant + across token steps (causality), so loop once + cached generation is + bit-identical and ≥3.5× faster. Latent planning is prefill-shaped + (parallel, compute-dense) — its economics improve with model scale, + unlike serial CoT decode. +- **Cross-token carry (design C)** is stable and the first recurrent variant + to beat its own baseline on math overall (12.9% vs 10.2%), via latent pause + positions, but does not beat the weights control; p=6 ≤ p=2 (no + latent-depth scaling). +- **Fixed operational traps** documented in `../LESSONS.md`: plan-pass + truncation masquerading as "planning hurts"; stop-string matching the + opening fence (pass@1=0 artifact); full-vocab logits OOM (earlyoom silent + kills); KV-cache append in layer re-runs; small-pool overfitting ~step 300; + val-CE checkpoint selection does not track generation accuracy. + +## Status of caveats + +Resolved: post-hoc k (pre-registered k=2 before unified test numbers); +weights-vs-loop attribution (FF arms); seed dependence (3 seeds, wave 5 adds +2 more); same-harness baselines. In progress tonight: outcome-selection on +hard buckets (sampled relabeling); equal-FLOPs explicit-planning reference; +depth generalization beyond trained k; lens battery at N=20; **12B +replication** (the scale question). Not addressed: second synthesis domain, +base-model diversity beyond gemma, easy-dip elimination.