132 lines
7.3 KiB
Markdown
132 lines
7.3 KiB
Markdown
# Prototype plan: the self-paced workspace (v2)
|
||
|
||
*Drafted 2026-07-16, pre-registration-style. Goal: test whether the model
|
||
can learn to allocate workspace-loop compute ON ITS OWN — per prompt and
|
||
per generation step — rather than at a swept hyperparameter k.*
|
||
|
||
## The concept
|
||
|
||
At every step the system chooses: emit, or spend a band iteration updating
|
||
the workspace first. Make that choice a learned gate g(workspace state).
|
||
Compute becomes a decision, not a constant. Gate-bought iterations emit no
|
||
tokens, so they are exposure-safe by construction (deterministic given
|
||
state — the pause-position property).
|
||
|
||
## What already exists (de-risked ingredients)
|
||
|
||
| ingredient | evidence | where |
|
||
|---|---|---|
|
||
| static per-prompt gate (E0) | gated 52.0 overall, easy 97.5 (vs 88.5 uniform), hard 28.6; bottleneck = probe recall (18/28 tp, 95 predicted hard) | `gate_probe.py`, `eval_gated.json` |
|
||
| state-dependent control heads train | adaptive-α rescued 12B (3.8K params) | AdaptiveMergeAdapter |
|
||
| evolving state during generation | carry beats registers on GSM (hard 0→9.4) | `carry_common.py`, `eval_carry.json` |
|
||
| loop-capacity knob | rung-2 band-LoRA = best hard numbers (42.9/46.4) | `lora_band.py` |
|
||
| gates over DEPTH (complementary axis) | learnable compute envelope g[t,l] | `path_gates.py` (Nils, in progress) |
|
||
| free gate supervision | STaR difficulty labels; per-position labels derivable | prep_star/prep_mbpp |
|
||
|
||
## Experiments
|
||
|
||
### E1 — learned per-prompt halting (prompt side, MBPP)
|
||
Replace fixed k with a trained soft halting gate. Architecture: after each
|
||
iteration i, gate head h(e, ŝ_i) → p_halt,i (zero-init to fixed-k
|
||
behavior); training uses the soft mixture of iteration outputs weighted by
|
||
halting distribution (ACT-style), CE + λ·E[iterations] compute penalty;
|
||
deploy = argmax halt. Trains end-to-end, NO RL. Arms: λ ∈ {1e-3, 1e-2},
|
||
vs E0 probe-gate and uniform-k anchors.
|
||
**Pre-registered predictions:** (a) accuracy ≥ uniform k=4 overall at ≤60%
|
||
of its mean iterations; (b) easy ≥ 95% (gate protects the substrate);
|
||
(c) allocation correlates with STaR label (point-biserial r > 0.3);
|
||
(d) hard ≥ E0's 28.6% (learned gate beats frozen probe recall).
|
||
**Failure mode to watch:** gate collapse (all-0/all-1) — mitigate with
|
||
penalty warmup + entropy bonus; collapse at all λ falsifies E1.
|
||
|
||
### E2 — generation-side gating (GSM, gated carry)
|
||
Substrate: design-C carry + short verified-CoT supervision (dense targets;
|
||
harvest with "solve in ≤3 short steps", answer-verified). Gate per token
|
||
step decides whether the carry state updates through the band or passes
|
||
through: x_t = g·merge(e_t, s_{t−1}) + (1−g)·e_t, penalty λ·E[g].
|
||
Anchors: carry-always, carry-never (same supervision).
|
||
**Predictions:** (a) gate fires non-uniformly, concentrated near numeric/
|
||
operator tokens (measurable); (b) accuracy ≥ carry-always (gating as
|
||
protection); (c) easy-bucket damage < carry-always's (83→45 was the
|
||
unprotected number). Hard-bucket *gain* over carry-always is hoped for,
|
||
not predicted.
|
||
|
||
### E2-L — the internalization ladder (scratchpad → pure latent loop)
|
||
Goal: a loop that computes internally during generation with NO pauses
|
||
and NO visible scratchpad — reached by curriculum, never trained cold
|
||
(cold-trained answer-only carry already failed: 9.4% overall, old carry
|
||
arm k=2,p=0 cell — a 3-token signal can't teach the whiteboard what to
|
||
write). Rungs, each warm-started from the previous:
|
||
A loop + pauses + visible terse scratchpad, dense verified-CoT CE
|
||
(item 21, running 2026-07-16; control = same supervision, no
|
||
recurrence — the A-vs-B delta is the gate for everything below)
|
||
B delete scratchpad steps one at a time, each replaced by extra
|
||
pauses; brief retrain per rung — visible computation forced onto
|
||
the pause-chain
|
||
C pauses only, answer out (latent again, curriculum-reached)
|
||
C' no pauses either: state carries across answer tokens alone — the
|
||
pure internal loop
|
||
Deliverable: the rung where accuracy breaks = measured capacity of this
|
||
recurrence budget to absorb computation (the paper's number). Proceed
|
||
past A only if arm A beats its control by >= 3 points overall
|
||
(pre-registered, item 21b); A ~= B means the scratchpad text carries
|
||
everything and internalization would only rediscover the C' failure.
|
||
|
||
### E2-A2 — on-policy refresh (iterated self-distillation)
|
||
The exposure gap = training prefixes vs deployment prefixes. Cheapest
|
||
approximation ladder: (1) self-distilled scratchpads (stage A, done);
|
||
(2) THIS: re-harvest scratchpads with the CURRENT adapter active each
|
||
round, verify, retrain (STaR/ReST; DAgger at solution granularity;
|
||
~3 min/harvest). Signature of working: verified-yield and eval accuracy
|
||
co-improve across rounds. Gated on the A-vs-B verdict.
|
||
|
||
### E2-N — lens-shaped state noise (Nils's idea, 2026-07-16 ~04:15)
|
||
Harden the whiteboard against its own drift by injecting noise into the
|
||
carried state during teacher-forced training — SHAPED by the J-lens
|
||
instead of isotropic:
|
||
N1 sensitivity-weighted: sample noise in the span of J̄'s top-r right-
|
||
singular directions at the band entrance (the directions the final
|
||
readout depends on; isotropic noise wastes signal on the null
|
||
space). Cheap: jbar.pt exists; --lensnoise rank,scale flag.
|
||
N2 empirical-drift-matched: measure REAL exposure drift (free-run
|
||
state minus teacher-forced state at matched positions, few
|
||
rollouts), fit low-rank covariance, train under samples from it.
|
||
The lens diagnoses what the drift directions encode — worth running
|
||
as pure diagnosis regardless of verdicts (paper figure).
|
||
N3 concept-jitter: lens-read the carried concept (e.g. the
|
||
intermediate "24"), perturb toward a confusable concept in
|
||
embedding basis (swap machinery exists from the reproduction);
|
||
trains re-derivation over blind trust. Most ambitious.
|
||
Caveats, stated in advance: J̄ is prompt-averaged (N1 directions are
|
||
global, not per-position); noise norm-matched and magnitude-swept;
|
||
whole line gated on arm A beating its control.
|
||
|
||
### E3 — power knob (only if E1 or E2 shows clean gating)
|
||
Warm-start rung-2 band-LoRA under the gate; joint fine-tune. Question: do
|
||
gate-bought iterations do MORE per iteration with a trainable band?
|
||
Metric: the internalization count (how many scratchpad steps can be
|
||
removed post-hoc, E2 curriculum) as a function of LoRA rank.
|
||
|
||
### Lens verification (throughout — our home advantage)
|
||
J-lens reads of gated vs ungated positions: do bought iterations sharpen
|
||
task-relevant concepts at the positions where the gate fired? This is the
|
||
mechanistic check that the gate allocates *meaningfully*, not just
|
||
correlationally.
|
||
|
||
## Explicitly out of scope for the prototype
|
||
Outcome-RL training of the gate (GRPO with compute price) — stage 2, only
|
||
if E1–E3 show selective gating. 12B/scale transfer. Cross-task gates.
|
||
|
||
## Budget & order
|
||
E1: 3 arms × ~75 min (Spark). E2: harvest ~30 min + 3 arms × ~90 min.
|
||
E3: +2 arms. Total ≈ 1.5 Spark-days. Runs after the lens campaign; queue
|
||
via gpuq as usual, every arm pre-registered in PROTOCOL_UNIFIED.md before
|
||
launch (items 18+).
|
||
|
||
## Kill criteria (decided in advance)
|
||
- E1 gate collapse at all λ AND E2 uniform firing → the state does not
|
||
carry usable "needs compute" signal at this scale; program stops, E0's
|
||
static-gate deployment note stands as the practical answer.
|
||
- E1 works but hard < E0 → learned gate worse than probe; ship probe-gate,
|
||
keep E2 only if its (a)/(b) hold.
|