three-paper split: plans, claim/evidence manifests, work plans

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

paper-A/PLAN.md

@@ -0,0 +1,63 @@
+# Paper A — Retrofitting Latent Planning onto Frozen LLMs
+
+**One-sentence claim.** A 1.6M-parameter merge adapter at an
+interpretability-chosen layer boundary gives a frozen LLM a silent
+("latent planning") mode that converts a large fraction of its
+plan-dependent failures into successes, at zero decode-time cost — and the
+gain is attributable to the recurrence, not the parameters.
+
+**Audience/venue.** Main-track ML (ICLR/NeurIPS class); fallback: strong
+workshop + arXiv. Base draft: `../PAPER.md` (restructure, don't rewrite).
+
+## Claims and evidence manifest
+
+| # | claim | evidence (files) | status |
+|---|---|---|---|
+| A1 | Hard-bucket transformation: 5.5%→37.4%±5.5, 5 seeds, full 500-item test | `results-loop/eval_code_{code_s0_full,code_s1,code_s2,code_s3,code_s4}.json` | ✅ |
+| A2 | Overall never below baseline; +1.3±0.8, 5/5 seeds ≥ (sign p≈.03) | same as A1 | ✅ |
+| A3 | Attribution: trained-noloop = untrained-loop = 17.9% ≪ trained loop | `eval_code_{ff,untrained,trained}.json` | ✅ |
+| A4 | Pause-token control (token-compute ≠ loop) | `eval_code_pause8.json` | ⏳ today |
+| A5 | Plan-distillation control (plan-info into weights ≠ loop) | `eval_code_distill.json` | ⏳ today |
+| A6 | Explicit-plan reference: loop recovers ~43% of gap at 0 tokens | `eval_plan_baseline.json` | ✅ |
+| A7 | Location matters (headline ablation only; full anatomy → Paper B): lens band 53.6/43.6 vs live wrong-bands ≤29/23.6; 12B clean version 32.8 vs 0.4–2.3 | `results-band-*/eval*.json`, `results-node-final/results-12b/eval_12b_gsm_band10*.json` | ✅ |
+| A8 | Depth generalization k=3/6/8 (curriculum can't fake) | `eval_code_code_kgen.json` | ✅ |
+| A9 | HumanEval transfer +8 overall; decomposition substrate +6.1 / training +1.3 | `eval_humaneval_{trained,untrained}.json` | ✅ |
+| A10 | Bucket integrity: sampled relabeling 97% agreement; per-bucket n + Wilson CIs everywhere | `mbpp_data.json` (`label_sampled`), per_item logs | ✅ (recompute in stats pass) |
+| A11 | Boundary section (math): no overall win at 2B or 12B; 4-arm grids; 12B attribution flip (loop>weights at scale); headroom law (gain tracks direct-vs-plan gap: 11→works, 8→works, 37→partial, 7→nothing) | `eval_{untrained,uni,ff,carry,pausectl}.json`, `results-node-final/results-12b/eval_12b_gsm_*.json`, `eval_code_12b_*` | ✅ |
+| A12 | Deployment: frozen-prompt KV write-in ≈2.9× prefill/0 decode; gate preserves easy items | `eval_gated.json`, equivalence probe (§ scripts) | ✅ |
+| A13 | Mechanism summary (2 paragraphs, cite Paper B): fixed point by k≈4; concept-probe sharpening 8×; KV-sharing caveat | `loop_dynamics.png` data, spider traces | ✅ |
+
+## Known weaknesses to state (not fix)
+- One base-model family (gemma-4); E2B + 12B only. 12B MBPP fails as-trained
+  (α miscalibration shown by untrained 23.0% — `eval_code_12b_untrained.json`);
+  reported as scale-sensitivity of α, with GSM-12B carrying the scale story.
+- HumanEval transfer is substrate-dominant; framed as such (no overclaim).
+- Lens-battery entropy metric uninformative (report honestly, keep concept-probe metric).
+- MBPP contamination paragraph (both arms share it; hard bucket conservative).
+
+## Missing / nice-to-have
+1. ⏳ A4/A5 land today (night chain).
+2. Blocksworld + Rust transfer (today): include as "task-generality" table row
+   whatever the outcome — flat/negative is fine, it feeds the headroom law.
+3. Optional strengthener (1 day, decide after draft): MBPP+ (EvalPlus strict
+   tests) robustness row for seed-0.
+4. NOT required: second model family, rung-2 — future work section.
+
+## Outline (9 pages)
+1. Intro — token-space vs latent serial compute; the retrofit gap; contributions.
+2. Background — J-lens & workspace (cite Paper D), recurrent-depth line.
+3. Method — band location; anchor-dominant merge; STaR labeling; k-curriculum;
+   prompt-only masking; frozen-prompt inference (fig: pipeline).
+4. Main results — A1/A2 (fig: 5-seed curves), A6, A9 (transfer), tables w/ CIs.
+5. Attribution — A3/A4/A5/A7/A8 grid (fig: attribution bars).
+6. Boundary — A11 (fig: 2×2 mechanism-placement grid, both scales).
+7. Mechanism summary — A13 (fig: dynamics + anchor cliff inset; details → Paper B).
+8. Deployment — A12; FLOPs framing.
+9. Related work, limitations, outlook (loopification-as-post-training thesis).
+
+## Work plan
+- [ ] Stats pass over all per_item JSONs (Wilson, McNemar, sampled-label recompute) — `scripts/stats_pass.py`, outputs `paper-A/tables/`.
+- [ ] Regenerate 5 figures into `paper-A/figures/`.
+- [ ] Restructure PAPER.md → `paper-A/draft.md` per outline; PDF.
+- [ ] User adversarial-review round; revise.
+- Owner of first draft: me, target: tomorrow evening.

paper-B/PLAN.md

@@ -0,0 +1,58 @@
+# Paper B — Where Retrofitted Recurrence Attaches: Injection Cliffs, Read-Invariance, and a KV-Sharing Hazard
+
+**One-sentence claim.** For loops retrofitted onto a frozen transformer, the
+write location is everything and the read location is nothing: performance
+falls off a one-layer cliff at the sensor→workspace boundary, is invariant to
+where the fed-back state is tapped, and — on KV-sharing architectures —
+becomes *structurally impossible* above the sharing boundary, a hazard that
+silently invalidates hook-based interventions far beyond this project.
+
+**Audience/venue.** Interpretability/analysis venue (e.g. BlackboxNLP-class
+workshop, or an interp track); alternatively TMLR. Cites Paper A for
+training recipe, Paper D for the lens.
+
+## Claims and evidence manifest
+
+| # | claim | evidence | status |
+|---|---|---|---|
+| B1 | Anchor cliff: inject L14→52%, L13→34%, L12→29%, L11→25% (monotone, one-layer onset at lens boundary) | `results-band-{13_30,12_30,11_30}/eval*.json` + reference | ✅ |
+| B2 | Tap invariance: taps 23/27/30/32/34 statistically flat (incl. motor/final layer); 40%-shorter loop body matches | `results-tap*/eval*.json` (note: 27/32/34 on H200, 23 on Spark, k0 anchors align) | ✅ |
+| B3 | KV-sharing hazard: E2B layers 15–34 reuse K/V from ≤14; band re-runs entered ≥15 are bit-exact nulls in training (zero grad) and behavioral nulls in eval (Δlogit up to ~11 with 0 argmax flips) | arm 17–27/24–34 logs; forensic probes (transcribe from session); `results-node-final/results-kvtest/train.log` | ✅ |
+| B4 | Causal demonstration: moving entrance one layer (14→15) reproduces the null on demand | `results-kvtest/train.log` (flat val@ all k) | ✅ (eval tail missing — rerun 5-min eval for completeness) |
+| B5 | 12B (no KV sharing) clean location test: wrong band 0.4–2.3% vs 32.8% | `results-node-final/results-12b/eval_12b_gsm_band10*.json` | ✅ |
+| B6 | Fixed-point dynamics: big first step, cos→1.000 by k≈4; explains accuracy/sharpening plateau and k-extrapolation flatness | dynamics data + `eval_code_code_kgen.json` | ✅ |
+| B7 | Discriminator: cliff = lens boundary vs last-full-attention-KV layer | **MISSING — the (9,30) arm** | ❌ queued idea |
+| B8 | Residual-only perturbation impotence: large logit shifts w/o attention pathway don't flip decisions | probe from session (rerun scripted for reproducibility) | ◐ script it |
+
+## The hazard section (the paper's service to the field)
+Write as a standalone, checkable warning: (1) detection recipe — check
+`num_kv_shared_layers`/equivalents before any layer-局部 intervention; (2)
+symptom catalogue — exact-baseline results, zero gradients, big-Δlogit/no-flip;
+(3) how our 3 pre-registered arms became silent nulls and how forensics found
+it (file diff → shared_kv_states kwarg → config). Affected model classes:
+KV-sharing efficiency variants (survey which public families have it).
+
+## Missing experiments (all cheap, Spark)
+1. **B7 (decisive): anchor (9,30)** — L9 is full-attention+KV-computing but
+   deep in sensor region. Pre-registration drafted in
+   `results-loop/PROTOCOL_UNIFIED.md` §(to add). Prediction on record: lens
+   boundary wins (arm 1 contained L9 and was catastrophic). ~70 min.
+2. B4 completeness eval (5 min).
+3. B8 as reproducible script + N=16 items (30 min).
+4. Optional: channel-count sweep (anchor 12 with tap 28 length-match) if B7
+   surprises. Decide after B7.
+
+## Outline (workshop-length)
+1. Setup: retrofit loop machinery (1 fig), what "write" and "read" are.
+2. Read-invariance (tap curve fig).
+3. Write-cliff (anchor curve fig + 12B clean replication).
+4. The KV-sharing hazard (anatomy fig: which layers recompute K/V; symptom
+   table; detection recipe).
+5. Fixed-point account tying 2–4 together.
+6. Implications for activation-intervention methodology.
+
+## Work plan
+- [ ] Run B7 + B4-eval + B8-script (one Spark session, ~2h GPU total).
+- [ ] Transcribe session forensics into `paper-B/hazard_notes.md` while fresh.
+- [ ] Figures: anchor curve, tap curve, KV anatomy diagram.
+- [ ] Draft after Paper A ships (it cites A's training recipe).

paper-D/PLAN.md

@@ -0,0 +1,51 @@
+# Paper D — The Verbalizable Workspace, Reproduced Across Scale and Architecture
+
+**One-sentence claim.** The J-lens/global-workspace findings reproduce on
+gemma-4 at 2B-effective, 12B dense, and 26B MoE — readouts and regime
+structure transfer robustly, exact averaged Jacobians are computable through
+a top-8 router — but the *write* basis migrates with scale
+(token-embedding writes at 2B, activation-derived vectors at 12B+), which
+the original account does not predict.
+
+**Audience/venue.** TMLR (reproduction/analysis) or ReScience; alternatively
+an interp workshop. Lowest-effort paper: `../RESULTS.md` is ~85% of it.
+Foundation citation for Papers A and B.
+
+## Claims and evidence manifest
+
+| # | claim | evidence | status |
+|---|---|---|---|
+| D1 | Readout reproductions (unspoken concept, cross-lingual, staging) at all three scales; strongest at 26B-MoE (spider P=1.0) | `results{,-12b,-26b}/heat_*.pt`, exp1 logs | ✅ |
+| D2 | Regime structure (transduction/sensor/workspace/motor) at all scales; fractional deepening at 12B; flatter/diffuse at MoE (J-space R²≈0) | `regimes*.pt`, exp4 logs, `results/regimes_montage.png` | ✅ |
+| D3 | Methodology: exact per-prompt Jacobians via batched VJPs; **exact through 128-expert top-8 routing** (identity anchor 0.0); costs documented | `scripts/compute_jacobians.py`, jbar checkpoints, timing logs | ✅ |
+| D4 | **Write-basis migration** (the novel finding): J-lens vectors read but never write; token-embedding writes 6/30+broadcast at 2B, 0/30 at 12B/26B; activation-derived vectors 30/30 at 12B and 26B — migration tracks scale, not architecture | `swap_grid.pt` (all three), exp2 logs | ✅ |
+| D5 | Directed modulation only clearly present at 26B (citrus P=0.69) | exp logs | ✅ |
+| D6 | Lens gate (where-to-intervene) transfers unchanged across all three | swap experiments | ✅ |
+
+## Missing / decisions
+1. **No new experiments required.** Optional strengthener: the un-run 31B
+   dense scaling point (~24–30h Spark or ~4–5h on a rented node) — decide
+   only if a reviewer asks; the 3-model story stands.
+2. Verify all claims against raw artifacts during rewrite (numbers in
+   RESULTS.md were written incrementally; re-derive tables from .pt files —
+   `scripts/stats_pass_d.py` to emit `paper-D/tables/`).
+3. Frame relationship to the original paper precisely: which claims
+   reproduce (readouts, regimes, gate), which do not (J-lens-basis *writes*),
+   and the migration as a boundary on the original's intervention story.
+4. Ethics/limitations: single model family; prompt-set sizes (150/100
+   prompts for J̄ at 12B/26B); MoE regime flatness may be estimator noise.
+
+## Outline
+1. Intro: why reproduce; what the workspace claims are.
+2. Methods: lens, exact VJP Jacobians (incl. MoE router handling), corpora.
+3. Reproduction results per scale (readouts, regimes) — montage figure.
+4. The write-basis migration (main figure: swap grids across scale).
+5. What transfers and what doesn't; implications (feeds Papers A/B).
+6. Repro details: costs, seeds, artifacts (all .pt files + bucket links).
+
+## Work plan
+- [ ] Re-derive all tables from artifacts (half day).
+- [ ] Rewrite RESULTS.md → `paper-D/draft.md` in reproduction-paper voice
+      (claims of original ⇄ outcome table up front).
+- [ ] Figures from existing .pt/pngs; regenerate montage at print quality.
+- [ ] Ship first — it is citable groundwork for A and B and needs no GPU.