52 lines
3.4 KiB
Markdown
52 lines
3.4 KiB
Markdown
# Paper D — The Verbalizable Workspace, Reproduced Across Scale and Architecture
|
||
|
||
**One-sentence claim.** The J-lens/global-workspace findings reproduce on
|
||
gemma-4 at 2B-effective, 12B dense, and 26B MoE — readouts and regime
|
||
structure transfer robustly, exact averaged Jacobians are computable through
|
||
a top-8 router — but the *write* basis migrates with scale
|
||
(token-embedding writes at 2B, activation-derived vectors at 12B+), which
|
||
the original account does not predict.
|
||
|
||
**Audience/venue.** TMLR (reproduction/analysis) or ReScience; alternatively
|
||
an interp workshop. Lowest-effort paper: `../RESULTS.md` is ~85% of it.
|
||
Foundation citation for Papers A and B.
|
||
|
||
## Claims and evidence manifest
|
||
|
||
| # | claim | evidence | status |
|
||
|---|---|---|---|
|
||
| D1 | Readout reproductions (unspoken concept, cross-lingual, staging) at all three scales; strongest at 26B-MoE (spider P=1.0) | `results{,-12b,-26b}/heat_*.pt`, exp1 logs | ✅ |
|
||
| D2 | Regime structure (transduction/sensor/workspace/motor) at all scales; fractional deepening at 12B; flatter/diffuse at MoE (J-space R²≈0) | `regimes*.pt`, exp4 logs, `results/regimes_montage.png` | ✅ |
|
||
| D3 | Methodology: exact per-prompt Jacobians via batched VJPs; **exact through 128-expert top-8 routing** (identity anchor 0.0); costs documented | `scripts/compute_jacobians.py`, jbar checkpoints, timing logs | ✅ |
|
||
| D4 | **Write-basis migration** (the novel finding): J-lens vectors read but never write; token-embedding writes 6/30+broadcast at 2B, 0/30 at 12B/26B; activation-derived vectors 30/30 at 12B and 26B — migration tracks scale, not architecture | `swap_grid.pt` (all three), exp2 logs | ✅ |
|
||
| D5 | Directed modulation only clearly present at 26B (citrus P=0.69) | exp logs | ✅ |
|
||
| D6 | Lens gate (where-to-intervene) transfers unchanged across all three | swap experiments | ✅ |
|
||
|
||
## Missing / decisions
|
||
1. **No new experiments required.** Optional strengthener: the un-run 31B
|
||
dense scaling point (~24–30h Spark or ~4–5h on a rented node) — decide
|
||
only if a reviewer asks; the 3-model story stands.
|
||
2. Verify all claims against raw artifacts during rewrite (numbers in
|
||
RESULTS.md were written incrementally; re-derive tables from .pt files —
|
||
`scripts/stats_pass_d.py` to emit `paper-D/tables/`).
|
||
3. Frame relationship to the original paper precisely: which claims
|
||
reproduce (readouts, regimes, gate), which do not (J-lens-basis *writes*),
|
||
and the migration as a boundary on the original's intervention story.
|
||
4. Ethics/limitations: single model family; prompt-set sizes (150/100
|
||
prompts for J̄ at 12B/26B); MoE regime flatness may be estimator noise.
|
||
|
||
## Outline
|
||
1. Intro: why reproduce; what the workspace claims are.
|
||
2. Methods: lens, exact VJP Jacobians (incl. MoE router handling), corpora.
|
||
3. Reproduction results per scale (readouts, regimes) — montage figure.
|
||
4. The write-basis migration (main figure: swap grids across scale).
|
||
5. What transfers and what doesn't; implications (feeds Papers A/B).
|
||
6. Repro details: costs, seeds, artifacts (all .pt files + bucket links).
|
||
|
||
## Work plan
|
||
- [ ] Re-derive all tables from artifacts (half day).
|
||
- [ ] Rewrite RESULTS.md → `paper-D/draft.md` in reproduction-paper voice
|
||
(claims of original ⇄ outcome table up front).
|
||
- [ ] Figures from existing .pt/pngs; regenerate montage at print quality.
|
||
- [ ] Ship first — it is citable groundwork for A and B and needs no GPU.
|