J-lens workspace reproduction + loop retrofit: lens, band looping, adapters, controls, multi-task evals

Reproduction of the 2026 workspace/J-lens paper on gemma-4 (E2B/12B/26B),
plus the workspace-loop retrofit line: merge adapter, prompt-only latent
planning (MBPP), carry variant, attribution controls (FF/pause/untrained),
band-location ablation, Blocksworld harness, 12B replication scripts.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

LESSONS.md

@@ -0,0 +1,78 @@
+# What worked and what didn't — workspace looping on gemma-4-E2B
+
+Lab-notebook distillation of the looping investigation (July 2026). Full
+results and figures: [`WORKSPACE_LOOPING.md`](WORKSPACE_LOOPING.md); base
+reproduction: [`RESULTS.md`](RESULTS.md). Everything on `google/gemma-4-E2B-it`
+(frozen), DGX Spark GB10.
+
+---
+
+## Scientific: what WORKED
+
+| finding | evidence | where |
+|---|---|---|
+| **Merge layer makes the band a stable recurrence** — anchor-dominant `(1−α)e + α·ŝ`, α=0.3; the whole failure of naive looping is that the band is not a self-map | untrained: answer preserved 11+ loops, converging (probe 2d) | `loop_common.py` MergeAdapter |
+| **Training the merge (1.6M params, model frozen) turns hold into amplify** | J-lens concept sharpens 8× across loops (0.015→0.13; untrained flat) | `train_merge.py`, eval JSONs |
+| **The loop is a true fixed-point iteration** — big first step, bit-exact convergence by k≈3–4; explains why accuracy & sharpening plateau there, k>4 is a no-op | cos(s_k, s_{k−1}): 0.926 → 1.000 | `plot_loop_dynamics.py` |
+| **Prompt-only looping ("latent planning") beats baseline on code** — first overall win: MBPP pass@1 52.0% vs 48.8%; plan-dependent problems 3.6%→46.4% (untrained control 14–18%; p<1e-5) | 250 test items, executed | `eval_loop_code.py`, §5 |
+| **Frozen-prompt KV-cache trick** — looped prompt states are constant across token steps (causality), so loop once, hook-swap the band input at prefill, generate cached. Bit-identical, ≥3.5× faster | equivalence test match=True | `generate_frozen_prompt` |
+| **STaR self-labeling gives difficulty labels + in-distribution supervision for free** — direct pass vs CoT/plan pass, verified answers/tests | GSM8K 16%/53%, MBPP 52%/63% | `prep_star_data.py`, `prep_mbpp.py` |
+| **Difficulty→depth curriculum is absorbed** — hard items (trained only at k≥2) fit best at depth | val CE gradient; hard-bucket peaks at k=4 | train logs |
+| **Untrained cross-token carry is stable** — band state carried across token steps, one pass/token; coherent 60+ tokens, mild norm drift; mean/EMA seeding ≥ last-position | `probe_carry.py` output | §6 |
+| **Trained sharpening transfers to the carry regime on short answers** — P('spider') 0.015→0.12 in ~2 token steps, amortized | carry probe w/ trained adapter | §6 |
+
+## Scientific: what FAILED (and what each failure taught)
+
+| failure | numbers | lesson |
+|---|---|---|
+| **Naive band loop** (L30 out → L14 in) | collapses in 1 iteration | the band is not a self-map; out-space ≠ in-space |
+| **Additive anchor** `H0 + bandΔ(H)` | diverges, ‖ΔH‖/‖H‖≈2/step | forward displacement double-counts; need interpolation, not addition |
+| **Frozen single tied layer** | stable but degenerate attractor; concept never sharpens | untrained layers tolerate looping, never benefit (matches Relaxed-Recursive) |
+| **α ≥ 0.6 merge** | drifts to garbage | the anchor must dominate; the loop is a perturbation, not a replacement |
+| **Full-position looping + teacher forcing (GSM8K)** | overall acc never beats k=0; easy items 97%→~35% in free-run while teacher-forced val looked great | exposure bias: generated tokens through a loop trained only on gold tokens. Loop placement must match deployment. Prompt-only looping fixed this *structurally* |
+| **Trained (within-token) adapter in the carry regime, long outputs** | tag-soup / token garbage after ~1 sentence | adapters must be trained on the trajectory they produce; short-answer training does not license long free-running recurrence |
+| **Uniform loop depth on mixed difficulty** | easy items pay ~6–13 points at k≥1 even trained | looping problems that need no extra compute only adds perturbation → per-prompt gate (probe on k=0 workspace state; STaR labels = free supervision) is the top v2 item |
+
+## Operational pitfalls (each cost real time — don't re-hit)
+
+1. **"Planning hurts" was a truncation artifact.** Plan pass at
+   `max_new_tokens=380` scored 17% vs 54% direct — every sampled output ran
+   out of budget mid-plan, before any code. At 700 tokens + terse-plan prompt,
+   planning adds ~11 points. *Check truncation before believing any
+   "CoT hurts small models" result.*
+2. **`stop_strings="\n```"` matched the OPENING fence** (generations start
+   `\n```python`) → every generation halted at ~4 tokens → pass@1=0.000 at all
+   k, including k=0. The k=0 sanity anchor is what caught it. *Always have a
+   known-value row in every eval.*
+3. **Full-vocab logits are the memory monster.** (B, T, 262k) tensors:
+   batch 32 eval + concurrent training = machine-wide OOM (kernel killed the
+   desktop session). Then `earlyoom` (installed on DGX Spark, trigger-happy
+   with unified memory) SIGTERM'd a modest eval *silently* — no traceback,
+   log just stops. Fixes: `logits_to_keep=1` during generation, batch ≤16,
+   **GPU jobs sequential**, and a watchdog that notifies when a process
+   disappears. Check `journalctl | grep earlyoom` for silent deaths.
+4. **Re-calling decoder layers appends to the KV cache.** Capture-and-rerun
+   machinery must forward with `use_cache=False` and strip
+   `past_key_values` from captured kwargs, or re-runs see doubled keys.
+   Correctness anchor that caught it: manual band+suffix re-run must
+   reproduce the plain forward bit-exactly (0.0 max diff).
+5. **Small hard pools overfit fast.** 38 hard items: val CE rises from ~step
+   300 of 600, depth-ordering inverts by step 500. Snapshot checkpoints and
+   evaluate the pre-overfit one (we used step 399); ~200 steps suffice.
+6. **Background jobs must be `setsid`'d** or the harness/session restart
+   kills them mid-run. And `pkill -f <pattern>` will match your own launcher
+   shell if the pattern appears in its command line.
+7. **Zero-init adapter output layer ⇒ zero grads upstream at step 0** — on
+   `mlp[0]` this is expected (LoRA-B-style), not a bug; check the output
+   layer's grad instead.
+
+## The three design rules that emerged
+
+1. **Anchor-dominant merge** makes any residual-stream recurrence well-posed;
+   train only the merge.
+2. **Loop placement must match task structure**: prompt-only (static plan)
+   for generation tasks; full/carry (evolving state) for state-tracking —
+   and each must be *trained in the regime it deploys in*.
+3. **Verify with the lens, gate with the labels**: the J-lens picks the band,
+   measures whether loops compute, and diagnoses failures; STaR difficulty
+   labels supervise both the curriculum and (next) the adaptive-depth gate.