96 lines
4.5 KiB
Markdown
96 lines
4.5 KiB
Markdown
# Interim report: workspace-loop retrofit, through wave 3
|
||
|
||
2026-07-13/14 · gemma-4-E2B-it (frozen throughout) · DGX Spark
|
||
All eval numbers: held-out test splits, greedy decode, execution/answer-verified.
|
||
Full protocol: `PROTOCOL_UNIFIED.md`; methods: `../WORKSPACE_LOOPING.md`.
|
||
|
||
## Headline finding
|
||
|
||
A 1.6M-parameter merge adapter (0.03% of the model) at the L13→L14 boundary,
|
||
trained ~30 min on ~280 self-labeled items, lets the frozen model **loop its
|
||
workspace band (L14–30) over the prompt k times before writing code** —
|
||
converting plan-dependent failures into successes:
|
||
|
||
**MBPP full test set (500 items), hard bucket (55 plan-only items), k=4:**
|
||
|
||
| seed | hard pass@1 | overall |
|
||
|---|---|---|
|
||
| baseline (k=0) | 5.5% | 51.8% |
|
||
| seed 0 | **43.6%** | 53.6% |
|
||
| seed 1 | **41.8%** | 53.8% |
|
||
| seed 2 | **30.9%** | 51.8% |
|
||
|
||
Mean hard-bucket effect ≈ 7× baseline; overall at-or-above baseline for all
|
||
seeds (no net tax). Convergent mechanism evidence: the trained loop is a hard
|
||
fixed-point iteration (cos→1.000 by k≈4), and the J-lens shows latent-concept
|
||
sharpening across iterations (~8× over untrained control on the probe task).
|
||
|
||
## Attribution: the loop is the ingredient (code), not the weights
|
||
|
||
Same data, same parameter count, same boundary — only the mechanism varies
|
||
(MBPP 250-item subset, hard bucket):
|
||
|
||
| arm | hard pass@1 |
|
||
|---|---|
|
||
| baseline | 3.6% |
|
||
| untrained loop (α-merge only) | 17.9% |
|
||
| trained NO-loop adapter (weights control) | 17.9% |
|
||
| trained loop | **42.9–46.4%** |
|
||
|
||
The trained feedforward control lands exactly on the untrained-loop number:
|
||
~18% is what perturbation + format alignment buys; the remaining ~28 points
|
||
require the recurrence.
|
||
|
||
## The boundary: math inverts the picture
|
||
|
||
No recurrent variant beats plain weights on GSM8K. Four-arm grid (hard bucket):
|
||
|
||
| GSM8K hard | prompt-side only | touches generation |
|
||
|---|---|---|
|
||
| feedforward weights | **11.8%** (best) | 4.7% (pause control) |
|
||
| recurrence | 6.3–8.7% (prompt-loop) | 9.4% (carry, beats FF 2× in-harness) |
|
||
|
||
Two orthogonal effects: (1) perturbing free-running generation positions is
|
||
costly for either mechanism; (2) recurrence beats weights only where a state
|
||
must evolve (generation side), and loses where it doesn't (static prompt).
|
||
Interpretation: the loop performs **plan refinement**. Code needs a plan
|
||
(loop wins); math needs answer-time computation that one frozen band pass per
|
||
step cannot supply at 2B (CoT tokens remain load-bearing). Overall GSM
|
||
accuracy: no variant beats the 10.5% baseline. (Hard-bucket cells carry an
|
||
outcome-selection caveat — greedy labels; sampled relabeling in progress —
|
||
so the math story rests on the overall numbers.)
|
||
|
||
## Secondary findings
|
||
|
||
- **Unified (mixed-task) training regressed both tasks** vs dedicated
|
||
adapters (MBPP 46.8% vs 52.0% at pre-registered k=2; GSM no better).
|
||
CE-level parity between arms did NOT predict generation parity.
|
||
- **Gated deployment** (logistic probe on the k=0 workspace state, STaR
|
||
labels as supervision): 52.0% overall with easy items fully preserved
|
||
(97.5%); gate recall 64%, precision 19% — the current ceiling; threshold/
|
||
feature work is CPU-only follow-up.
|
||
- **Frozen-prompt KV-cache equivalence**: looped prompt states are constant
|
||
across token steps (causality), so loop once + cached generation is
|
||
bit-identical and ≥3.5× faster. Latent planning is prefill-shaped
|
||
(parallel, compute-dense) — its economics improve with model scale,
|
||
unlike serial CoT decode.
|
||
- **Cross-token carry (design C)** is stable and the first recurrent variant
|
||
to beat its own baseline on math overall (12.9% vs 10.2%), via latent pause
|
||
positions, but does not beat the weights control; p=6 ≤ p=2 (no
|
||
latent-depth scaling).
|
||
- **Fixed operational traps** documented in `../LESSONS.md`: plan-pass
|
||
truncation masquerading as "planning hurts"; stop-string matching the
|
||
opening fence (pass@1=0 artifact); full-vocab logits OOM (earlyoom silent
|
||
kills); KV-cache append in layer re-runs; small-pool overfitting ~step 300;
|
||
val-CE checkpoint selection does not track generation accuracy.
|
||
|
||
## Status of caveats
|
||
|
||
Resolved: post-hoc k (pre-registered k=2 before unified test numbers);
|
||
weights-vs-loop attribution (FF arms); seed dependence (3 seeds, wave 5 adds
|
||
2 more); same-harness baselines. In progress tonight: outcome-selection on
|
||
hard buckets (sampled relabeling); equal-FLOPs explicit-planning reference;
|
||
depth generalization beyond trained k; lens battery at N=20; **12B
|
||
replication** (the scale question). Not addressed: second synthesis domain,
|
||
base-model diversity beyond gemma, easy-dip elimination.
|