Files
jspace/results-loop/PROTOCOL_UNIFIED.md
T

985 lines
62 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Pre-registered protocol: unified adapter eval (written before any test numbers)
Date: 2026-07-13, after val@499, before step-799 completion. No test-set
number for the unified adapter exists at time of writing.
1. **Primary loop depth: k=2, for both tasks.** Chosen on val CE with the
tie-breaking rule: prefer the SMALLEST k whose hard-cell val CE is within
0.01 nats of the best k. (Current val: MBPP hard k2k4 = 0.004, GSM hard
k2k4 = 0.006 → both ties → k=2.) The full k-curve is secondary/descriptive.
2. **Checkpoint selection criterion (scalar, fixed now):** mean of the two
hard-cell val CEs at k=2, tasks weighted equally:
crit = (gsm_hard_k2 + mbpp_hard_k2)/2. Lowest crit among saved checkpoints
wins. (At writing: step 499, crit = (0.475+0.205)/2 = 0.340.)
3. **Primary endpoints:** (a) MBPP test pass@1 hard-bucket at k=2 vs k=0;
(b) GSM8K test accuracy hard-bucket at k=2 vs k=0. McNemar, paired by item.
Overall accuracy is secondary (known to be underpowered at n=250/256).
4. **Same-harness rule:** all k, INCLUDING k=0 baselines, measured by the
prompt-only fast-path scripts (`generate_frozen_prompt`; k=0 = plain
cached generate inside the same function). No numbers carried over from
the full-position-loop harness.
5. **Known missing control (not covered by this run):** a same-size,
no-recurrence adapter (h -> h + MLP(h) at the L13->L14 boundary, no loop,
no band re-run) trained on identical data/objective. Until it exists,
"the loop does the work (vs. 1.6M new weights anywhere doing it)" is NOT
established. Queued as the next training run. Note the k=0 column is
gated off by construction and is a sanity check only — it is not this
control.
6. **Symmetric interference check (missing):** dedicated GSM8K prompt-only
adapter as the reference for "unified costs GSM nothing". Queued. Until
then the no-interference claim is one-directional (MBPP side only).
7. **Band-location ablation (pre-registered 2026-07-13, before any arm ran).**
Arms, all else identical (adapter size/init, data, curriculum, k, scripts;
MBPP): early L2-12, mid-narrow L17-27, late L24-34 (width-matched, 11
layers); shifted L6-22 (width-matched to the original 17). Reference:
workspace L14-30 (already run, 3 seeds). Prediction: workspace-centered
arms (L14-30, L17-27) exceed early/late on hard-bucket pass@1 at k=2-4 by
a wide margin; shifted intermediate. Falsification: near-parity across
arms demotes the lens claim from "locates where to loop" to "convenient
discovery tool"; to be reported either way. Primary readout: hard-bucket
pass@1 at k=4, e400 checkpoints throughout.
8. **Language commitments for the writeup:** the k0->k1 CE collapse (e.g.
4.36->0.18) is format/template learning expected from any trained adapter
and must not be quoted as evidence of routing/planning; informative
comparisons are within k>=1 cells only. Depth ordering k2 vs k4 deltas
(0.001-0.006 nats) are inside checkpoint jitter and must be described as
"k>=2 fits hard items equally well; k=1 slightly worse."
9. **Anchor/entrance sweep (pre-registered 2026-07-14 ~03:00, before any arm
ran).** Arms: bands (13,30), (12,30), (11,30) — injection point shifted
up from L14 at fixed tap L30; plus tap-23 = (14,23). All E2B/MBPP, same
recipe, e400, primary readout hard-bucket pass@1 at k=4. Competing
predictions: (a) "L14 special" (last full-attention KV-computing layer,
lens boundary) → anchor-13 drops; (b) "KV-channel count" (anchors 11-13
add 1-3 extra KV-recomputing attention channels) → holds or improves.
Body-length confound noted: earlier anchors lengthen the loop body; if
results shift, run matched-length control (12,28) before interpreting.
10. **L9 discriminator arm (pre-registered 2026-07-14 ~10:00, before running).**
Band (9,30): anchor at L9 — the only other full-attention, KV-computing
layer below the boundary — deep in the lens's sensor regime, tap fixed at
L30. Separates the two cliff explanations: (a) "lens boundary" predicts
catastrophic (like anchors 11-13: 25-34% overall); (b) "full-attention
KV-layer entry" predicts partial recovery (clearly above the L11-13
trend, i.e. >40% overall or hard >25%). Registered prediction: (a) —
the sensor-region content dominates; layer type does not rescue it.
Same recipe/checkpoint/eval as the anchor sweep (250 items, ks 0,2,4).
---
# Outcomes vs pre-registrations (scored 2026-07-14, after all arms completed)
1. **k=2 primary depth** — held. All primary comparisons reported at k=2;
k-curves descriptive. k≥2 plateau confirmed (k=8 gen-eval flat).
2. **Checkpoint criterion** — applied as written for the unified adapter.
Separately reported: val-CE is a poor proxy for generation accuracy;
later arms therefore pre-committed to fixed steps (e400) instead.
3. **Primary endpoints (unified adapter, k=2 vs k=0)** — (a) MBPP hard
3.6% → 28.6% (direction as predicted); (b) GSM hard 0% → 6.3%, overall
10.5% → 9.0% (no overall win — the math boundary result). Both reported.
4. **Same-harness rule** — held throughout (all final tables fast-path,
k=0 included).
5. **Missing weights control** — run: trained FF adapter = 17.9% hard,
exactly the untrained-loop level. Loop-vs-weights gap established.
6. **Symmetric interference check** — run (dedicated GSM adapter);
mixed-task training regressed both tasks; reported as negative result.
7. **Band-location ablation** — prediction CONFIRMED with a caveat:
L14-30 hard 43.6% ≫ early L2-12 (23.6%, overall destroyed 22.8%) and
shifted L6-22 (21.8%, overall 29.0%). Caveat discovered: L17-27 and
L24-34 are structurally null (KV sharing; k>0 ≡ k=0 bit-identical), so
the "mid-narrow beats late" half of the prediction was untestable at
E2B; the 12B replication (no shared KV) carries that weight instead.
8. **Language commitments** — honored in PAPER.md (k0→k1 CE collapse not
cited as planning evidence; k2-vs-k4 nats described as jitter).
9. **Anchor/entrance sweep** — prediction (a) "L14 special" CONFIRMED:
anchor-13 hard 17.9%/overall 34.4%; 12: 28.6%/30.8%; 11: 25.0%/25.2%;
monotone collapse below the boundary. Tap-23 arm died in training
(never rerun); exits 27/30/32/34 within seed noise, so exit choice is
free. Matched-length control not needed (results did not shift with
body length in the informative direction).
10. **L9 discriminator** — registered prediction (a) CONFIRMED: band
(9,30) overall 14.0-21.4%, hard ≤21.4% — catastrophic, like anchors
11-13, despite L9 being a full-attention KV-computing layer. The lens
boundary, not layer type, gates the retrofit.
11. **Recurrent-regime arm (pre-registered 2026-07-15, before training).**
Huginn-style retrofit on the frozen E2B band: RecurrentAdapter
(learned A,B init α·I/(1−α)·I + zero-init MLP), h0 = norm-scaled
noise, log-uniform random depth k∈[1,16], bptt=4, same data/steps/
checkpoint rule (e400 primary) as all merge arms. Eval ks 0,2,4,8,16,32
on the 250-item MBPP set. Competing predictions: (a) "amortization is
intrinsic to frozen-band retrofits" → performance plateaus by k≈4 at
or below the merge arm's level, no depth-monotone gain; (b) "fixed-
point behavior was an artifact of our fixed-shallow-k training"
(Huginn regime transfers) → monotone hard-bucket improvement past k=8
and reduced noise-seed sensitivity after training. Secondary readout:
path independence (two noise seeds → output agreement rate) at e400.
Known risk, stated in advance: 600 steps may be far too little for
this regime (McLeish et al. use ~50B tokens); a null here bounds the
cheap-retrofit budget only, not the regime.
12. **Parcae-constrained recurrent arm (pre-registered 2026-07-15, before
training; Prairie et al. 2026 parameterization).** Same as item 11 but
A = exp(−Δt·exp(a)) diagonal → ρ(A) < 1 by construction; init exactly
the α=0.3 merge (verified bit-equal at init). ρ(A) logged every 10
steps in BOTH arms. Theory-derived predictions, stated in advance:
(a) contraction ⇒ fixed point is a function of e ⇒ the Parcae arm
SATURATES in k (no depth-monotone gain) and its converged performance
is amortizable — if so, our deflationary result is a corollary of
ρ<1, and our observed k≈34 convergence is the geometric rate 0.3^k;
(b) the UNCONSTRAINED item-11 arm either drifts toward ρ≥1 (watch the
ρ log: divergent runs should show ρ≥1 before loss spikes) or, if it
gains monotone depth-performance, does so with ρ near 1 — the edge of
stability is where genuine iteration must live. Either outcome
formalizes "the anchor coefficient is the stability dial" as
"the anchor coefficient is the spectral radius".
13. **Per-depth adapter arm + free-ACT probe (pre-registered 2026-07-15,
before training).** (a) PerDepthAdapter: one merge adapter per
iteration (n=4, Bae-style depth-wise relaxation at the entrance;
breaks time-invariance — LTV, no fixed-point guarantee), standard
curriculum, e400, eval ks 0,2,4,8. Prediction: lands at or below the
distill/rung-2 amortization ceiling (~46% hard) because depth-indexed
weights add content, not state-evolution; exceeding it would show
per-iteration expressivity was binding and amend the deflationary
claim. Depths >4 reuse adapter 4 (stated: k=8 cell is then
fixed-point-like by construction). (b) Free-ACT probe on the standard
merge arm: record per-item convergence depth (cos>0.9995) at k=8 cap.
Predictions: accuracy unchanged vs fixed k (post-convergence no-ops);
mean k_conv ≈ 3; hard-labeled items converge SLOWER than easy ones
(adaptive compute allocates like ACT without any learned halting
parameter).
--- Outcome, item 11 (scored 2026-07-15, k=16/32 cells cancelled by
decision after k<=8): PREDICTION (a) SUBSTANTIALLY CONFIRMED, with one
twist. The unconstrained arm left contraction immediately (rho(A):
0.3 -> 3.4 by step 100, plateau ~4.5) yet trained smoothly — per-iteration
norm-matching converts magnitude explosion into directional churn, so
"rho>=1 => divergence" becomes "rho>=1 => divergence OR stationary churn"
under a norm projection. Consequences as predicted: substrate damage
(easy 98.4 -> ~69% at all k>0, far exceeding any contractive arm's tax),
val CE flat k=1..16 (stationary, not progressive), hard bucket at
merge level (35.7/39.3/42.9% at k=2/4/8 — a one-item-per-depth-doubling
crawl that at k=8 reaches what the contractive merge reaches at k=4,
never approaching the amortization ceiling from above). 4x parameters
bought nothing. Depth-monotone computation did not emerge at this budget.
14. **Tied-alpha arm (pre-registered 2026-07-15, before training).**
TiedAlphaAdapter: x = (1a)⊙e + a⊙ŝ + MLP([e;ŝ]), a = σ(â) per-dim
learned, init a=0.3 everywhere (bit-equal to MergeAdapter at step 0,
verified). B tied to (1a): convex combination keeps the LTI fixed
point on the e–ŝ segment (substrate-anchored by construction),
ρ = max(a) < 1 guaranteed, +d≈1.5K params. Standard curriculum,
s0 = band(e), e400, eval ks 0,2,4,8 on 250 items. This is the one
untested cell combining parcae's learnable decay with the merge's
anchoring. Predictions: (a) substrate fidelity preserved (easy ≈
merge's 88%, unlike both rec arms' ~70%) because anchoring, not
ρ, controls fidelity; (b) hard-bucket at merge level (no significant
gain — per-dim constant α is not where capability lives, per the
adaptive-α E2B result); (c) learned a drifts slightly DOWN from 0.3
(as in parcae). If (a) holds while rec arms failed it, the
fixed-point-location dial is causally isolated: same learnable-decay
freedom, only the tie to (1a) differs from parcae.
--- Outcome, item 12 (scored 2026-07-15): prediction (a) CONFIRMED in its
dynamics half, REFUTED in its fidelity half — and the refutation is the
finding. Dynamics: rho stayed in (0,1) throughout (0.300 -> 0.292, the
optimizer drifting MORE contractive when confined to the stable region);
loss trajectory as good as or better than the unconstrained arm at every
checkpoint (the rec arm's flight to rho~4.5 was epiphenomenal — all fit
lives in the MLP); eval saturates completely (hard 42.9/42.9/39.3/39.3/
39.3 at k=2/4/8/16/32, easy flat ~71%). Fidelity: easy items were NOT
preserved (71% vs the merge's 88.5%) despite guaranteed contraction —
substrate fidelity is controlled by fixed-point LOCATION (anchored B +
curriculum), not by rho. Conclusion: stability and fidelity are
independent dials (fig_phase.png); the Parcae constraint delivers exactly
what it promises (robust training, convergence, certified tail gradients)
and exactly nothing more. Item 14 (tied-alpha) is the causal isolation of
the fidelity dial.
15. **Fidelity factorial + capacity control + seed (pre-registered
2026-07-15 ~03:15, before any of these arms ran; overnight batch).**
The fidelity loss of both rec arms (easy 88.5 -> ~71%) confounds three
deltas from the winning merge: (i) learned B, (ii) random-depth
training instead of the difficulty->depth curriculum, (iii) noise s0.
Item 14 (tied-alpha) tests (i) with anchoring. New single-variable
cells, everything else = standard merge recipe (fixed B, band(e) s0,
curriculum, e400, eval ks 0,2,4,8 on 250 items):
a. merge+randk — only (ii) changed (log-uniform k in [1,16], bptt 4).
b. merge+noises0 — only (iii) changed.
c. merge h=2048 — capacity control for the per-depth arm (6.4M
shared vs 6.4M depth-indexed): if per-depth beats the ceiling
but h2048 does not, time-variation (not capacity) is credited;
if both do, it was capacity all along.
d. parcae seed 1 — robustness of the fidelity refutation.
Predictions: (a) and (b) each cost a few points of easy at most
(anchored fixed point dominates); neither reproduces the ~17-point
drop — the culprit is the learned/free B (with item 14 as the
positive control). h2048 stays at the ceiling (hard <=46%), fidelity
intact. parcae s1 reproduces easy ~71% within seed noise.
--- Outcome, item 13a (scored 2026-07-15): prediction CONFIRMED — per-depth
lands below/at the ceiling, never above. Detail is instructive: fidelity
preserved throughout (easy 88.5/89.3/86.9 at k=2/4/8 — anchored B), but
hard-bucket content is DEPTH-STRANDED: 17.9% at k=2 (adapters 3-4, which
hold the hard-trained content, never execute), 35.7% at k=4, 42.9% at k=8
— where depths 5-8 reuse adapter 4, i.e. the architecture reverts to
shared-map iteration and the fixed-point mechanism collects the remaining
gain. Time-variation adds a fragility (content unavailable except at its
training depth) and no capability; map-sharing is load-bearing for the
anytime-usable gain. Depth-4 adapter overfit visible in val (hard k4 CE
0.188@99 -> 0.371@599) — LTV concentrates small-pool overfitting into
single depths.
--- Outcome, item 13b (scored 2026-07-15): accuracy prediction CONFIRMED
(k=8 halt run 52.0/90.2/42.9 = plateau level); convergence predictions
REFUTED. Per-item state-cosine (thresh 0.9995, k=8 cap): k_conv
distribution 4:3, 5:57, 6:47, 7:17, never-within-8:126 — mean ~7, and NO
difficulty gradient (easy 7.01 vs hard 7.00). The earlier "bit-exact by
k~3-4" was the single dynamics-probe example, not the population: outputs
plateau by k~2-4 while the state keeps drifting at 1e-3..1e-4 cosine
scale; the fixed point is an OUTPUT-stable orbit (suffix layers + decode
wash out residual state motion), not a literal state fixed point for most
prompts. Free-ACT via state-cosine therefore yields no early exit at this
threshold, and no ACT-like difficulty allocation falls out for free —
output-level halting signals would be needed. Paper's dynamics claims
softened accordingly.
--- Outcome, item 14 (scored 2026-07-15): ALL THREE PREDICTIONS CONFIRMED.
(a) Fidelity fully preserved: easy 93.4/91.0/90.2 at k=2/4/8 (merge:
92.6/88.5; parcae with identical decay freedom but untied B: ~71%) —
the free B is causally isolated as the fidelity culprit, the anchoring
tie as the protection. (b) Hard at merge level exactly (35.7/42.9/39.3 =
merge's k-curve within noise); no gain from the freedom. (c) Learned a
essentially unmoved: mean 0.298, range [0.285, 0.310], 0/1536 dims moved
>0.05 from init — the anchor coefficient is not a useful learnable DOF;
hand-tuned 0.3 was already optimal. Recipe consequence: fixed-alpha
anchored merge is the recommended design; learnable-alpha safe but
pointless, learnable-B harmful, per-depth strands the gain.
16. **Code→GSM8K cross-task transfer (pre-registered 2026-07-15 ~14:10,
before running).** The MBPP-trained loop adapter (adapter_code, s0) and
the noise-s0 variant evaluated on GSM8K test (n=256, prompt-only loop,
same harness as eval_gsmonly). Extends the transfer-distance ladder
(HumanEval tie -> LCB trained-hurts) across tasks. Predictions:
(a) hard-bucket gain ~0 (plan content is task-local; GSM8K needs
evolving state, not static plans); (b) easy items damaged at k>0
(~93 -> 50-70%), comparable to or worse than the GSM-trained merge —
substrate damage on GSM8K is perturbation-driven and content-agnostic;
(c) overall at k>0 below k=0 (no rescue). If instead hard gains
appear (>5 points), plan-shaped content is partially task-general —
would weaken the task-local claim from LCB.
--- Amendment to item 15 (2026-07-15 ~13:15): noise-s0 arm EXCEEDED
prediction (b) upward: hard 50.0/53.6/50.0 at k=2/4/8 with easy 88-90%
— nominally the best hard cells of the project (merge best 46.4; seed
mean 37.5±5.5). Paired vs tied-alpha (only same-day per-item baseline):
discordants 5-1/3-0/3-0 in noise-s0's favor, each k p≈0.22-0.25 at n=28
— consistent direction, not individually significant. Denoising
interpretation: training the loop to reach the fixed point from noise
regularizes the content. SEED ARMS QUEUED (s1, s2, same recipe/eval,
pre-registered here): if seed-mean hard(k=4) > 46.4 (the merge's best
single cell), the recommended recipe gains noise-s0; if seed mean falls
back into 37-46, it was a lucky seed.
--- Outcome, item 15c (h2048 capacity control, scored 2026-07-15): the
per-depth exoneration is CLEAN — shared 6.4M params reach hard 42.9/53.6/
50.0 at k=2/4/8 vs per-depth's 17.9/35.7/42.9 at the same capacity;
time-variation is strictly worse than weight-sharing at matched params.
Fidelity prediction confirmed and exceeded (easy 95.1% at k=2 — best
looped fidelity of the project; 90.2% at k=4/8). Ceiling prediction
(hard <= 46%) REFUTED UPWARD like noise-s0: k=4/8 at 53.6/50.0. Two
independent variations (noise s0, 4x MLP) now sit at 50-54% where the
original merge reached 46.4 — suggests 46.4 was an UNDER-estimate of the
recipe family's level, not a ceiling it defined. The distill-parity
deflation claim is unaffected statistically (53.6 vs 45.7 at hard n=28
is within noise) but the language "every regime tops out at the same
ceiling" should become "at the same level within noise" — pending the
noise-s0 seed arms.
--- Outcome, item 15d (parcae seed 1, scored 2026-07-15): CONFIRMED —
the fidelity refutation replicates. easy 70.5/73.0/72.1 at k=2/4/8
(seed 0: 72.1/71.3/70.5); hard 32.1/39.3/35.7 (seed 0: 42.9/42.9/39.3,
ordinary seed spread at n=28). Two-seed conclusion: contraction-with-
free-B loses ~17 points of easy items regardless of seed; the phase
diagram's Parcae point is solid.
--- Outcome, item 16 (code->GSM8K transfer, scored 2026-07-15): ALL THREE
PREDICTIONS CONFIRMED, emphatically. MBPP-trained loop on GSM8K: hard
0.8-1.6% at every k (prediction a: ~0 gain — plan content is task-local);
easy 93.1 -> 27.6-44.8% (prediction b: damaged, in fact WORSE than the
GSM-trained merge's 48%); overall strictly below k=0 at every k>0
(prediction c). noise-s0 variant identical (easy 34.5, hard 1.6). The
transfer-distance ladder ends cleanly: near (HumanEval) tie, far-code
(LCB) trained-hurts, cross-task (GSM8K) trained-content actively toxic
while gaining nothing. Task-locality of the learned content is now a
three-point monotone result.
--- Closure of the item-15b/15c "ceiling nudged upward" question
(2026-07-15, after ns seeds): LUCKY SEED, per the pre-registered rule.
noise-s0 hard(k=4) across seeds: 53.6 / 39.3 / 35.7 -> seed mean 42.9,
inside the 37-46 band. Fidelity across seeds intact (easy 90.2-94.3 —
the factorial conclusion is seed-robust); the 50-54% cells (ns seed 0,
h2048 single seed) were upper-tail draws of the same distribution the
merge's 46.4 came from. No recipe amendment; the abstract's original
"same level within noise" framing stands; single-cell records are not
levels — only seed means are.
17. **GSM-only, current recipe (pre-registered 2026-07-15 ~20:45, before
running).** train_merge_unified.py --tasks gsm: MergeAdapter, prompt-
only loop, curriculum, GSM8K data ONLY — removes the mixed-task
interference confound from the adapter_uni run, completing the
"winning recipe trained on GSM" question. Eval: prompt-only, n=256,
ks 0,1,2,4, e400. Predictions: (a) hard <= 10% at every k (supervision
density is structural: ~3 answer tokens; the recipe's dense-output
ingredient cannot exist here); (b) easy damaged at k>0 (to 40-70%);
(c) overall never beats k=0. If hard exceeds 15% or overall beats
k=0, task interference in the mixed run was masking a real GSM
capability — would reopen the GSM chapter.
Scope note (item 17): the design-space arms of items 11-15 are NOT
crossed with GSM8K, deliberately. Exclusion by dominance: fidelity-
failing regimes (rec, parcae) cannot improve on a task MORE fidelity-
fragile than MBPP; architecture-failing (per-depth) and equivalent
(tied-alpha -> merge) and k-placement-only (randk) and same-family
(noise-s0, h2048) variants have no mechanism by which task change
could invert their MBPP verdict. Only the recipe family's best member
(this item) is informative on GSM8K.
--- Outcome, item 17 (GSM-only, current recipe, scored 2026-07-15):
predictions (a) and (b) CONFIRMED, (c) nominally exceeded but not
meaningfully. hard 8.7/5.5/4.7% at k=1/2/4 (below the 10% bar; nowhere
near the 15% reopen threshold); easy 93.1 -> 48-52% at k>0; overall
11.7/10.9/10.2 vs k0's 10.5 — the k=1 cell is +1.2 points nominal
(~3 items at n=256, not significant), the rest below. Removing the
mixed-task interference bought ~2 points over adapter_uni (9.4 -> 11.7
at k=1) — interference was real but marginal, not masking a capability.
The GSM8K chapter is closed: the recipe family's best member, trained
on GSM alone in the correct regime, delivers no usable gain and the
standard fidelity damage; combined with the scope note, the boundary
claim (structural: supervision density + state-evolution bottleneck)
is fully supported.
18. **E1: learned per-prompt halting gate (pre-registered 2026-07-16
~00:20, before any arm runs; PLAN_SELFPACED.md).** HaltingMergeAdapter:
frozen-recipe merge + ACT-style halting head on the last prompt
position's workspace state; soft state-mixture training, CE + lambda *
E[iters], penalty warmup at step 100; NO difficulty curriculum (mixed
batches — the gate must discover the allocation). k_max=4, e400/e600
checkpoints, deploy = sequential halting at 0.5 cumulative mass,
generation via frozen-prompt at per-item k*. Arms: lambda in
{0, 1e-3, 1e-2}, seed 0. Eval: 250 items, vs anchors k=0 (0.488),
uniform merge k=4 (0.512/0.885/0.464), probe-gate E0 (0.520/0.975/0.286).
Predictions: (a) some lambda gives overall >= 0.512 at mean E[k] <=
2.4 (60% of uniform-4); (b) easy >= 0.95 at that lambda; (c) k*-vs-hard
point-biserial r > 0.3; (d) hard >= 0.286 (beats E0's frozen probe).
Collapse (E[k] pinned at 1 or 4 for all lambda) falsifies E1 and
triggers the plan's kill criterion. lambda=0 control isolates whether
the CE gradient alone moves the gate (expected: barely — penalty
provides the pressure).
Item 18 amendment (2026-07-16 ~23:45, before results): arms run on a
rented 4xH100 node in parallel instead of the Spark queue; a fourth
arm (lambda=1e-3, seed 1) is added for immediate seed replication of
the expected-winner penalty. Spark's queued gate jobs will be dropped
to avoid duplication. Everything else per registration.
--- Outcome, item 18 (scored 2026-07-16 ~00:40): predictions (b), (c)
REFUTED, (a) marginal miss, (d) trivial pass. All arms converge to
UNIFORM depth (lambda 0/1e-3/1e-2 -> E[k] 4/2-or-4/1; the two 1e-3 seeds
picked different plateaus — degenerate penalty landscape), r = 0.000
everywhere. Mechanism identified and consistent with prior findings:
teacher-forced CE is depth-flat (stationarity), so CE provides no
per-item depth gradient; the penalty alone cannot teach selectivity.
The state DOES carry the signal (E0 probe: train acc 1.0) — the failure
is the training signal, not the representation. E1-as-designed is dead;
kill criterion NOT fully triggered (E2 untested, and the mechanism
points at a repair).
19. **E1b: label-supervised halting head (pre-registered 2026-07-16
~00:45, before running).** Freeze the curriculum merge (adapter_code
s0); train ONLY the halting head (BCE): target halt=0 at iterations
below the label's depth (easy->1, hard->4, per STaR label), halt=1 at
or above it. 300 steps, mixed batches, head-only params. Eval: gated
eval as item 18, n=250. Predictions: (a) r(k*, hard) > 0.5 (the head
is a trained difficulty classifier now); (b) easy >= 95% at k*=1
(near-E0's 97.5); (c) hard >= 35.7% (>= best uniform arm, via better
recall than E0's frozen probe: more than 18/28 hard items routed
deep); (d) overall >= 52.0 at E[k] <= 2.2. If (c) fails while (a,b)
hold, halting-head recall saturates at probe level and gate quality,
not gate training, is the binding constraint.
--- Outcome, item 19 / E1b (scored 2026-07-16 ~01:15): prediction (c)
CONFIRMED (hard 39.3 >= 35.7 at mean k* 2.18), (a) FAILED at r=0.217
(selectivity real — hard routed 2x deeper than easy (2.18 vs 1.08), the
program's first nonzero gate correlation — but weak at deploy), (b,d)
FAILED for a traced design reason: halted_k_per_item lacked k*=0, so easy
items were forced through >=1 iteration and landed on the merge's WORST
easy depth (k=1: 85.2%); E0's 97.5% came precisely from k=0 routing.
E1c amendment (pre-registered before running, same session): pre-loop
halt consult on s_0 enabling k*=0; targets easy->0, hard->4; threshold
0.5 unchanged (calibration deferred unless E1c misses). Predictions:
easy >= 95%, hard >= 35.7%, r >= 0.4, overall >= 51.2 at E[k] <= 1.5.
--- Outcome, item 19 / E1c (scored 2026-07-16 ~01:50, Spark re-run):
prediction (b) CONFIRMED — easy 95.9% with mean k*=0.11 (the k=0 routing
fix worked perfectly for easy items); (a) FAILED (r=0.220, unchanged);
(c) FAILED HARD — hard 21.4% at mean k*=1.5: with the pre-loop halt
consult, weakly-discriminated hard items now exit at k*=0/1 where before
they got >=1 iteration; the recall problem became more expensive, not
better. (d) FAILED (overall 50.0 at E[k]=0.74). Net Pareto: E1c =
(50.0 overall, 95.9 easy, 21.4 hard, 0.74 mean iters — 82% compute
saved); E0 probe-gate = (52.0, 97.5, 28.6, ~2.2) still dominates on
accuracy. The learned-head line has ONE identified un-tried knob:
deploy-threshold calibration on val for hard-recall (the 0.5 threshold
is arbitrary; raising it routes more items deep, trading easy tax for
hard recall — a tunable curve E0 cannot offer). E1 arc summary for
PLAN_SELFPACED: gating machinery works end-to-end, easy-item protection
and compute savings are demonstrated and cheap; difficulty-selective
DEPTH allocation remains unsolved at 3K-param-head scale — binding
constraint is classifier quality on the k=0/s0 state, exactly where E0
started. Next per plan: threshold sweep (cheap) before any E2.
20. **E1 threshold curve + oracle bound (pre-registered 2026-07-16 ~02:15,
before running).** Phase 1: record E1c head's halt probabilities per
test item (one GPU pass). Phase 2: per-item outcomes for the frozen
curriculum merge at k=0/1/2/4 (four generation sweeps, tag merge_lut —
doubles as the reusable gate-evaluation lookup table and supplies the
long-missing per-item logs for the canonical merge). Phase 3 (offline):
gated accuracy at thresholds .3-.99 by composing k*(theta) with the
lookup; plus the ORACLE gate (best k per item) = the ceiling any gate
can reach with this merge. Predictions: (a) some theta gives hard >=
32% with easy >= 93% and E[k] <= 2.2 (dominating E0 on compute at
comparable accuracy); (b) the curve is monotone in theta; (c) oracle
overall >= 55% — if so, gate-quality headroom is large and further
gate work is justified; if oracle < 53%, gating this merge is nearly
saturated and the program pivots to E2 or closes.
--- Outcome, item 20 (scored 2026-07-16 ~02:35): (b) CONFIRMED — clean
monotone threshold curve (hard 7->50%, easy 96.7->87.7%, E[k] 0.43->2.63
across theta .3->.99). (a) FAILED — no theta reaches easy>=93 AND
hard>=32; at matched easy the E0 frozen probe dominates the entire
learned-head curve: the BCE-trained 3K head is strictly worse than the
class-balanced logistic probe it was meant to replace. (c) CONFIRMED,
emphatically: ORACLE gate = 59.6 overall / easy 100% / hard 64.3% at
E[k]=0.24. Key insight: hard items are DEPTH-DIVERSE — 18/28 solvable at
some k in {0,1,2,4} but no single k solves more than 13; a third of the
hard bucket lives in per-item depth selection. Program continues per
rule; binding constraint quantified: gate quality is worth ~9.6 overall
points (50.0 deployed vs 59.6 oracle). Also noted: the LUT re-run of the
canonical merge shows small systematic drift vs the Jul-13 eval (k4 hard
46.4 identical, k1/k2 hard 3 items lower) — the LUT (per-item, single
harness run) is now the canonical reference. Next candidates, in cost
order: (i) deploy E0's probe AS the gate against the LUT (free,
offline); (ii) stronger classifier (multi-position features, more data,
calibrated threshold); (iii) oracle-gap error analysis on the hard items
no fixed k solves but some k does.
21. **E2 stage A: dense short-CoT supervision through the carry
whiteboard, GSM8K (pre-registered 2026-07-16 ~02:55, before running;
PLAN_SELFPACED E2 / the hybrid from the internalization discussion).**
Prep: harvest TERSE verified CoTs ("at most 3 short steps", answer-
verified, STaR filter) for GSM train. Arms: (A) carry regime
(k=2 prefill, pauses easy p=2 / hard p=6) trained with CE on
scratchpad+answer (~30-60 dense tokens — the ingredient every latent
GSM arm lacked); (B) CONTROL: identical supervision, feedforward
adapter, no recurrence. Eval: GSM test 256, grid 0:0 (base), 2:2,
2:6; e400 checkpoints. Predictions: (a) arm A beats every previous
GSM arm's overall (>12.1%) — dense supervision is the binding fix;
(b) the A-vs-B delta isolates the whiteboard: if A > B by >=3 points
overall, recurrence adds value beyond visible-scratchpad training;
if A ~= B, the scratchpad text alone carries it (deflation, GSM
edition); (c) easy-bucket damage smaller than answer-only carry's
(83->45%) because training and deployment output formats now match.
Honest note: arm outputs are VISIBLE tokens (~40) — this is the
budget-CoT-with-loop hybrid, a scope change from latent planning,
run at Nils's explicit direction ("do gsm8k and such").
--- Outcome, item 21 (scored 2026-07-16 ~06:30). Harvest: 427 verified
terse CoTs (292 hard, 135 easy; 73.5% yield, ~3 min). Grid (n=256,
base 10.9/93.1/0.8/0.0 overall/easy/hard/drop):
arm A carry 2:2 53.9 (82.8 / 60.6 / 37.0)
arm A carry 2:6 57.4 (72.4 / 63.8 / 45.0)
control FF 2:2 49.6 (79.3 / 59.8 / 28.0)
control FF 2:6 54.7 (82.8 / 66.9 / 31.0)
(a) CONFIRMED, dramatically: best cell 57.4% vs the previous best GSM
arm's 12.1% — a 5x jump to full-CoT territory (~53%) at ~1/4 the
visible tokens. Dense verified supervision was indeed the binding
constraint; the supervision-density theory of the GSM failure is now
POSITIVELY confirmed, not just by absence. (b) MIXED: A-vs-B delta
+4.3 at p=2 (clears the >=3 bar), +2.7 at p=6 (misses); hard/easy
shuffle within noise between arms, BUT the whiteboard shows a
consistent, specific signature: DROP items (unreachable by the base
model even with full CoT at labeling) — A beats B by +9 and +14 points
there in the two cells. Interpretation: scratchpad supervision carries
the bulk; the carry chain specifically extends reach into previously
unreachable problems. McNemar (scored 2026-07-16 morning): overall
A-vs-B not significant in either cell (2:2 discordants 34-23 p=0.185;
2:6 33-26 p=0.435), but the pre-identified drop-bucket signature IS:
2:6 drop discordants 20-6, exact McNemar p=0.0094 (survives
Bonferroni x4 = 0.038); 2:2 drop 16-7, p=0.093 (same direction,
marginal). Verdict: the whiteboard's edge is real and specific to
extending reach into drop items, not a general lift over matched
supervision. (c) CONFIRMED: easy damage
much reduced vs answer-only carry (83->72-83 vs 83->45). Ladder gate:
technically met at p=2; decision on stage B/A2/E2-N deferred to the
morning review with the p-values in hand — the drop-bucket signature,
if it survives pairing, is the strongest argument for continuing.
22. **E2-L rung B: internalization ladder, front-first step deletion
(pre-registered 2026-07-16 ~10:40, before running; Nils chose stage B
over A2/E2-N at the morning review. Gate state: pre-registered +3
overall met on points (+4.3 at p=2); paired McNemar overall ns, but
the drop-bucket signature significant, 2:6 p=0.0094).** Design:
delete the first d scratchpad lines of each verified terse CoT
(d=1,2,3 — front-first: the deleted computation must ride the
pause-chain before the visible remainder), each deleted step replaced
by 10 pauses (median step = 10 tokens, compute-matched); unparseable
cots (14/427 without exactly one Answer line) pass through intact.
Step-count distribution 1/2/3/4/5+: 11/164/196/27/15 — so d=3 is
effectively rung C (pauses only) for ~87% of items. Each rung
warm-starts from the previous (d=1 from rung-A e400), brief retrain:
200 steps, LR 3e-4 cosine, seed 0. Eval: GSM test n=256, cells 0:0
(k=0 sanity, expect ~base 10.9), 2:(2+10d), 2:(6+10d); e200
checkpoints; per-item logs kept so rung-vs-rung McNemar is offline.
Known approximation, stated in advance: items with fewer than d
steps train at smaller effective p than the eval cell
(ndel=min(d,n_steps)). Predictions: (a) d=1 best cell within 5
points of rung A's 57.4 — one step fits the recurrence budget (the
drop-bucket reach evidence says the whiteboard already carries
step-sized computation); (b) monotone decline across d; (c) at d=3
accuracy stays above BOTH base (10.9) and cold answer-only carry
(9.4) — curriculum beats cold training even where the ladder breaks.
Deliverable: the break rung = first d whose best cell falls >=5
points below the previous rung's best — the measured capacity of
this recurrence budget to absorb computation. Job:
scripts/jobs/zzz_m_gsm_rungb.sh (single submit, ~3x(40min train +
eval) on the Spark).
--- Outcome, item 22 (scored 2026-07-16 ~15:30). k=0 sanity row
reproduced base exactly in all three rung evals (10.9/93.1/0.8/0.0).
Ladder (best cell overall, n=256): rung A 57.4 -> d=1 31.6 (2:12) ->
d=2 18.4 (2:26) -> d=3 19.1 (2:32). (a) FAILED, decisively: the break
rung is d=1 — deleting ONE compute-matched step costs 25.8 points
(easy 83->62, hard 64->36, drop 45->17); the recurrence cannot absorb
even one step's computation at this budget/recipe. (b) monotone
through d=2, then a plateau (d=2 18.4 -> d=3 19.1, within n=256
noise): the decay bottoms out ABOVE the floor rather than collapsing
to it. (c) CONFIRMED: d=3 (pause-only for ~87% of items) = 19.1 vs
base 10.9 — paired McNemar 33-12 discordants, p=0.0025 — and vs cold
answer-only carry 9.4: the curriculum-reached latent loop DOUBLES the
cold-trained equivalent. Composition of the d=3 edge: easy is
DAMAGED (58.6 vs 93.1 base) while hard (18.9 vs 0.8) and drop (8.0
vs 0.0) are lifted — the latent loop trades easy-bucket reliability
for reach, echoing item 21's drop-bucket signature in latent form.
Deliverable: measured capacity of this recurrence budget = a plateau
at ~19% overall / ~2x the cold floor; the visible scratchpad carried
the other ~38 points. Val-loss note: hard val rose with d (0.36 ->
0.56 -> 0.59) while easy fell (d=3 0.13, mostly answer-line targets)
— CE fit does not track eval accuracy at deep rungs. Interpretation
for the plan: rung C' (no pauses) is moot as a capability claim — C
already plateaued at 19; the E2-L line's value is now the 2x-over-cold
curriculum effect + the reach trade, not scratchpad-free parity. Next
knobs if the line continues: longer per-rung retraining, finer
deletion schedule (fractional rungs), or E2-N noise-hardening stacked
on the rungs.
23. **E2-L d=1 capacity-ceiling controls (pre-registered 2026-07-16
~15:55, before running; Nils: "run that one control").** Item 22's
break-at-d=1 has two untested confounds; one arm each, single-knob
changes from item 22's d=1 (both warm-start rung-A e400, seed 0):
arm A "x600" = 600 steps instead of 200 (3x training; tests the
brief-retrain objection — note d=1 val had plateaued at 0.36, so
prediction is NO recovery); arm B "pp30" = 30 pauses per deleted
step instead of 10 (3x latent bandwidth, same 200 steps; tests
whether inert-pause capacity, not training, binds). Eval n=256:
arm A cells 0:0, 2:12, 2:16 (e600); arm B cells 0:0, 2:32, 2:36
(e200). Decision rule, stated in advance: item 22 d=1 best = 31.6;
a control within +-5 points confirms the ceiling on that axis;
>=+5 (>36.6) reopens the ladder on that knob (longer schedules or
wider pause-chains); if BOTH land within +-5, the d=1 ceiling is
confirmed structural and the ladder chapter closes as scored.
Job: scripts/jobs/zzz_n_rungb_ctrl.sh.
AMENDMENT (2026-07-16 ~19:05, mid-run, Nils's call): arm A's eval
skipped to save ~50min GPU — the verdict was already decided by
training evidence (train loss memorized to 0.02-0.10 by step 200;
val easy 0.386->0.397->0.426, hard 0.357->0.376->0.421 across
e200/e400/e600 — monotone UP, textbook overfit, no recovery; k=0
sanity row reproduced base before the kill). Arm A scored from the
val trajectory: training time is NOT the binding constraint. Arm B
(pp30) runs in full as registered.
--- Outcome, item 23 (scored 2026-07-16 ~23:55; rc=0). k=0 sanity rows
reproduced base in both arms' evals (arm A's before its eval was
skipped). Arm A (x600): scored from val trajectory per amendment — no
recovery, mild overfit; training-time axis CONFIRMS the ceiling. Arm B
(pp30): best cell 29.3 (2:36; 28.5 at 2:32) vs item-22 d=1's 31.6 —
WITHIN the +-5 band, slightly below: 3x latent positions bought
nothing (hard 33.9/36.2 vs 36.2; drop 14 vs 17; easy 51.7-55.2 vs
62.1). Decision rule: BOTH axes confirm -> the d=1 break is
STRUCTURAL. Neither longer training nor a longer pause-chain lets
this carry absorb one scratchpad step; the binding constraint is the
carried state itself (its per-position expressivity/fidelity, not its
compute budget). Converging evidence from the same evening's
microscopy (probe_discount*/probe_gsm*): the board natively carries
plans ("Multiply", "subtract"), coarse magnitudes ("sixty/eighty"),
and completion-state ("plus" kept alive at the 430 divergence — carry
defers where FF commits a wrong digit at 99.3%), while exact digits
appear only just-in-time, 1-2 positions pre-emission. The ladder
chapter closes as scored in item 22; the loop program's live paths
are state-side (rung-2 band-LoRA / wider merge, E2-N hardening,
coarse-target auxiliary supervision matched to the medium) or the
hybrid (A2), not longer/denser pause-chains.
24. **E2-L d=1 with a trainable band: loop-only band-LoRA
(pre-registered 2026-07-17 ~00:30, before running; Nils: "can we
unfreeze the entire band and try this again?").** Item 23 located
the constraint in the carried state's per-position transformation;
this is the state-side attack. Full unfreeze rejected in design
(642M params vs 427 examples, and it would break the frozen-model
guarantee); instead LoopLoRA (lora_band.py, built for the rung-2
design): rank-16 deltas on q/v/down of EVERY band layer (L14-30,
uniform scale 1.0, 4.8M params), active ONLY during band re-runs —
initial forward and k=0 stay bit-exact by construction. Otherwise
identical to item-22 d=1: front-first deletion, 10 pauses/step,
warm-start rung-A e400 (adapter lr 3e-4; fresh LoRA lr 1e-3, B
zero-init so step-0 matches item 22 exactly), 200 steps, seed 0.
Eval n=256: 0:0 (sanity, must equal base), 2:12, 2:16; e200.
Decision rule (same bands as item 23, vs d=1's 31.6): >=+5
(>36.6) = band expressivity was binding -> escalate (rank 64, all
projections, or full-band rung ladder); within +-5 = the medium
verdict stands even with a trainable band -> internalization line
CLOSED, program pivots (A2 / E2-N / coarse-aux / divergence
batch). Prediction, honest: mild gain but under threshold (~33-36)
— the microscopy says the board's failure mode is WHAT it carries,
not how flexibly it transforms it; but this is the experiment the
'wider state' hypothesis deserves before the line closes. Job:
scripts/jobs/zzz_p_rungb_blr.sh.
--- Outcome, item 24 (closed 2026-07-17 ~02:10, STOPPED by Nils
mid-eval: "stop that idea. does not seem to work either"). Training
completed; evidence at stop: val easy 0.346 / hard 0.405 vs
adapter-only 0.356/0.361 — the trainable band did not improve fit
(hard slightly worse), consistent with the registered prediction that
capacity/expressivity is not what binds. k=0 sanity row with trained
LoRA loaded reproduced base EXACTLY (10.9/93.1/0.8) — the loop-only
LoopLoRA guarantee holds in practice; the mechanism is validated even
though the hypothesis died (useful for any future band-side work).
Accuracy cells not measured (eval killed at Nils's call to save ~50min
GPU); adapter+lora e200 checkpoints in the bucket if anyone ever wants
the number. Verdict: with items 22+23 this closes the E2-L
internalization line entirely — adapter capacity, training time,
pause bandwidth, and band expressivity have all now failed to move
the d=1 break. The carried state's native cargo (plans, magnitudes,
completion-state — see the probe series) is the program's remaining
asset; next candidates: divergence batch replay, coarse-target
auxiliary supervision, A2, E2-N.
25. **E2-L d=1 with latent process supervision through the lens
(pre-registered 2026-07-17 ~02:45, before running; Nils's idea:
"for training, i wonder if we could calculate, using jspace lens,
how each iteration should think").** Items 22-24 all trained the
latent chain blind — output CE only — and all failed; this changes
the INFORMATION reaching the chain, not its capacity. New loss:
the lens readout softmax(W_U·finalnorm(J̄_L30·h)) is differentiable
in h, so at the 10 replacement pauses we apply lens-CE against the
DELETED step's tokens, aligned 1:1 (pause j <-> step token j,
truncated at 10) — the board is trained to write the deleted step
in lens-readable code at the time it would have been written.
Mixed loss CE_out + λ·CE_lens. Two arms, single submit: λ=0.3 and
λ=1.0. Otherwise identical to item-22 d=1 (front-first deletion,
warm-start rung-A e400, adapter-only 3e-4, 200 steps, seed 0; no
band-LoRA — one knob). Smoke: step-0 lce=10.3 (~uniform: pauses
currently encode nothing about the step; large fresh gradient).
Eval n=256: 0:0, 2:12, 2:16 per arm. Decision vs d=1's 31.6, same
bands: >36.6 = latent supervision was the missing ingredient ->
ladder REOPENS with lens-taught rungs (and the 2D per-iteration
variant becomes item 26); within +-5 = even telling the board
exactly what to write doesn't make the carry compute it -> the
strongest closure evidence yet. Caveats pre-stated: J̄ is
prompt-averaged (global directions); the loss forces a
verbalizable code (microscopy suggests that IS the board's working
code, but a native non-verbal code would be fought); the 1:1
temporal alignment is one choice among several (bag-of-tokens,
result-digits-only are untested alternatives if this null's).
Job: scripts/jobs/zzz_q_rungb_lt.sh.
IN-FLIGHT NOTE (~03:20): λ=0.3 arm's lens-CE collapsed 10.3 -> 2.3
by step 40 — world 1 (cannot encode) is eliminated before the eval;
the open question is world 2 (writes but doesn't compute) vs
world 3 (reopens the ladder).
--- Outcome, item 25 (closed 2026-07-17 ~01:25, TRUNCATED by Nils:
"cancel that 2:16 and then try the new approach"). λ=0.3 arm: lens-CE
converged to ~1.9 (the pause-chain reliably writes the deleted step in
lens code, top-few over 262k vocab) while eval 2:12 = 31.2
(drop .18 / easy .586 / hard .354) vs the 31.6 reference — FLAT, not
even a redistribution across buckets. k=0 sanity clean. World 2
confirmed on the measured cell: WRITING IS NOT COMPUTING — the
carried state can be taught to verbalizably display the missing step
at the right timesteps and the displayed content is not consumed
downstream. 2:16 and the λ=1.0 arm cancelled mid-run (λ=0.3's flat
cell made λ=1.0 uninformative; ~2.5h GPU saved). Program value: the
cleanest representation/computation dissociation in the series; the
read-side (what the suffix computation consumes from the board) is
now the prime suspect, and the pause-tape architecture allows
display-without-routing. Item 27 (zero-pause internal looping, no
tape to display on) runs next by Nils's direction; item 26 requeued
after it (job renamed zzz_t).
26. **Result-staging supervision during generation (pre-registered
2026-07-17 ~03:25, before running; Nils: "can we try to train the
loop using jspace during generation?").** The microscopy's
signature of the healthy board: each line's RESULT is on the
carried state before it is emitted ("1/twelve" at the '=',
P=0.997). This trains that signature directly. For every visible
scratchpad line, at the pre-'=' positions (the result is NOT yet
in the causal context there — nothing can be copied; low loss
requires computation) lens-CE the L30 carried state against the
line's result tokens; Answer-line analog via the ':' separator.
Spans computed in token space (gen_staging_targets; 1102 spans on
the d=1-transformed 427). Two arms, one submit, both otherwise
item-22 d=1 (warm-start rung-A e400, 200 steps, adapter-only):
(a) lg=0.3 alone — staging supervision without pause supervision;
(b) lt=0.3 + lg=0.3 — the full lens curriculum (deleted step
taught at the pauses, results staged during emission). Eval n=256:
0:0, 2:12, 2:16 per arm; decision vs 31.6, same bands as items
23-25. Distinct from item 25's question: 25 teaches WHAT the
silent chain should hold; 26 teaches the compute-before-emit
discipline on the visible chain (the behavior that separated
carry from FF at the 430 divergence). Caveat: on-policy ("during
generation" in the free-running sense) is NOT this — that's a
rollout-based follow-up (latent DAgger) if either lens arm moves.
Job: scripts/jobs/zzz_r_rungb_lg.sh.
--- Outcome, item 26 (scored 2026-07-17 ~10:50; ran last, requeued as
zzz_t after items 27/28 per Nils's mid-night reprioritization). k=0
sanity clean both arms. Arm a (staging-only, lg03): 24.2 / 23.4 —
the series' first ACTIVELY NEGATIVE result, ~7 points below the 31.6
reference, outside the band on the downside (easy 44.8-48.3, hard
~26-28, drop 12-15). Arm b (combined lt03+lg03): 15.6 / 14.1 —
compounding destruction (easy 27.6, hard 18-21, drop 5-6), worse than
either constraint alone. Reading: the microscopy showed the healthy
board computes results 1-2 positions pre-emission (just-in-time); the
staging loss demanded the result across the ENTIRE pre-'=' span,
including positions where operands haven't been read — forcing
thought earlier than the computation can exist corrupts the schedule
rather than accelerating it. Lesson for any future lens supervision:
match the loss's temporal demand to the board's measured schedule
(supervise at the '=' +- 1 position only), don't blanket-force.
Overnight program complete: items 25-28 all scored; write-side
exhausted; the standing decision (clamp test vs pivot to hybrid/A2)
is unchanged by this result.
27. **Internal band looping, zero pause tokens (pre-registered
2026-07-17 ~01:10, before running; Nils's clarified structure,
option A confirmed via question: one silent burst before output).**
Architecture: NO pause tokens anywhere — sequence is prompt +
visible output only. After the k=2 prompt settle, the last prompt
position runs M=10 IN-PLACE band iterations (s^i seeds from the
position's own previous band output — carry_common inplace
updates), then generation proceeds with the normal single-pass
carry under the visible tokens. Vertical thought burst instead of
a horizontal pause tape: iterations leave no KV entries — only
the final state survives, a true internal loop (the C'-flavored
architecture the ladder never tested; every prior rung bought
compute with positions). Supervision: lens-CE iteration i <->
deleted-step token i (λ=0.3, the stable value), same d=1 deletion,
warm-start rung-A e400, 200 steps, seed 0. Eval n=256: 0:0
(sanity), 2:0 with --inner-iters 10 (matched), 2:0 with
--inner-iters 0 (ablation: does the burst matter at inference?).
References: positional d=1 31.6 (same ±5 / >36.6 bands) and the
ii0 ablation cell (isolates the burst's causal contribution).
Honest priors: item 25's 2:12 just landed at 31.2 (flat), so the
write-vs-compute dissociation likely carries over; the
architectural deltas that could matter here: no attention-tape
(forces state-borne computation rather than KV re-reading) and
anchor-at-prompt (iterations see the full settled question).
Job: scripts/jobs/zzz_s_rungb_ii.sh.
IN-FLIGHT NOTE (~02:20): lce 9.4 -> 2.9 by step 60 — the tape-free
burst also encodes the step trajectory; encoding is never the
obstacle. Accuracy pending.
--- Outcome, item 27 (scored 2026-07-17 ~04:35). k=0 sanity clean.
Matched cell 2:0 (burst on) = 34.0 (drop .19 / easy .621 / hard .394)
— nominally the BEST d=1 number of the series (vs 31.6 positional,
31.2 lens-pauses) with ZERO pause tokens; ablation (same adapter,
burst off) = 32.4. But both pre-registered comparisons are
noise-compatible when paired: burst-on vs burst-off 10-6 discordants,
p=0.45; vs positional d=1 44-38, p=0.58. Verdict: within the +-5
band — the ceiling holds on the internal-iteration axis too; the
burst's causal contribution is directionally positive (all of it in
the hard bucket, +3.2) but unproven at n=256. Notable anyway: the
zero-pause FORMAT loses nothing (ablation 32.4 >= positional 31.6 with
12 fewer latent positions) — the pause tape was contributing ~nothing,
consistent with item 25's display-without-routing reading. Item 28
(teacher-state targets, same architecture) runs next and completes
the 2x2 (verbal/full-state x tape/no-tape).
28. **Teacher-state distillation into the burst (pre-registered
2026-07-17 ~02:30, before running; Nils's variant: "meaningful CoT
chunks yield internal state that we then teacher-force into the
loop-only model").** Items 25/27 supervise the VERBAL SHADOW of
the deleted step (token targets through the lens); this forces the
FULL state. Teacher = the frozen warm-start (rung-A e400) adapter
running the complete cot (deleted step visible), zero pauses, same
carry architecture; capture its band-exit state at the deleted
step's LAST token — "the state of having finished thinking the
step" — one no-grad pass over the 413 parseable items at startup
(targets fixed, no moving teacher). Student: identical zero-pause
M=10 burst as item 27, but the loss is cosine distance between the
burst's FINAL iterate s^10 and the teacher state, plus output CE.
Weight AMENDED pre-run λ=1.0 -> 5.0 after the smoke measured the
starting cosine distance at only 0.113 (nearby band-exit states
share most structure); at 1.0 the term would be ~10x weaker than
the CE and a null would be underpowered — 5x0.113 puts the two
terms at comparable initial scale. No lens loss (one knob vs item 27: full-state
targets instead of verbal-shadow targets; trajectory supervision
dropped — only the endpoint is forced). Known approximation,
stated in advance: teacher state is captured ~10 positions later
in the sequence than the student anchor (RoPE position coloring);
cosine + the carry machinery's routine state transplantation
across nearby positions make this tolerable, but a null could
partly hide here. Eval n=256: 0:0, 2:0 ii10 (matched), 2:0 ii0
(ablation). Decision vs 31.6, same bands; the informative
three-way is 25 (verbal, tape) vs 27 (verbal, no tape) vs 28
(full state, no tape) — if 28 moves where 25/27 don't, the
computation-carrying content lives OUTSIDE the verbalizable
subspace; if none move, the read-side clamp test (item 29
candidate) decides. Job: scripts/jobs/zzz_sa_rungb_ts.sh.
--- Outcome, item 28 (scored 2026-07-17 ~06:20). k=0 sanity clean.
Distillation SUCCEEDED geometrically: cos-dist 0.113 -> 0.044 (burst
endpoint lands ~60% closer to the teacher state), with the lowest
output-CE tax of any supervised arm (vals 0.384/0.376). Function did
NOT follow: matched 2:0 = 30.1, ablation (burst off) = 29.7 — burst
contribution nil; hard 34.6 indistinguishable from the pack; and the
state-forcing DAMAGED easy (48.3 vs 62.1 in every other d=1 arm —
the forced endpoint displaces something easy handling needed).
Verdict: null, below reference. The 2x2 (items 25/27/28 + 22 as the
unsupervised corner) is complete and uniform: no state-side
supervision — verbal shadow or full teacher state, tape or burst —
makes the loop compute the deleted step. Combined with items 22-24
(capacity/time/positions/expressivity all null), the write-side is
exhausted end to end. The one untested link is READ-side: whether the
suffix computation consumes the workspace at all when the right state
is present — the clamp test (write the teacher state into the anchor
at inference via the lens/embedding-basis machinery, no training, and
measure) is item 29's natural content and would assign blame causally.
Nils's morning decision: clamp test vs pivot to the hybrid/A2 line
(where the 57.4 and the drop-bucket reach already live).
29. **Trajectory teacher-forcing of the burst (pre-registered
2026-07-17 ~11:50, before running; Nils's design: "measure the CoT
run's L30 at 10 evenly spaced intervals and teacher force them
into the loop — we know both the previous that goes into the merge
layer and the next L30 state").** The key upgrade over item 28:
the teacher trajectory decomposes the burst's learning into TEN
INDEPENDENT SUPERVISED TRANSITIONS — iteration i receives teacher
waypoint T[i-1] as its merge input and its band output is pulled
onto T[i] (cosine, λ=5.0) — dense gradient at every iteration, no
long BPTT chain, no compounding; and T[0] == the student's own
settled anchor state exactly (shared prompt+settle), so the chain
starts aligned by construction. Exposure gap handled by a second
arm adding the free-running loss (burst's own s_i onto T[i],
λ=5.0). The free-running burst ALWAYS runs last and seeds the
answer scan (training matches inference); TF transitions are
scaffolding erased from context afterward. Teacher = frozen
warm-start adapter on the full cot, 10 evenly spaced L30 states.
Two jobs, per Nils (both chosen): JOB 1 (zzz_u) d=1, waypoints
across the deleted step, arms tf-only and tf+fr; reference 31.6,
same bands; eval 0:0 / 2:0 ii10 / 2:0 ii0 per arm. JOB 2 (zzz_v)
THE HEADLINE: d=all (answer-only output, rung C), waypoints across
the FULL CoT — the burst as a 10-waypoint compressed rehearsal of
the entire reasoning; references: cold answer-only 9.4, curriculum
plateau 19.1, base 10.9; same arms and cells. Smokes: ltf/lfr
baseline 0.116 both modes; d=99 deletes 1118 steps across 427.
Priors, stated: the transition-decomposition is the first
mechanism that trains the loop's DYNAMICS rather than its
endpoints — if the loop can compute at all, this is its best
chance; if job 2 beats 19.1 meaningfully, compressed-trajectory
rehearsal becomes the internalization method. All prior caveats
(position coloring, teacher=warm-start quality) carry over.
Jobs: scripts/jobs/zzz_u_traj_d1.sh, scripts/jobs/zzz_v_traj_full.sh.
IN-FLIGHT, arm 1 (tf-only, d=1) scored ~15:40: THE FIRST
SIGNIFICANT POSITIVE OF THE PROGRAM, WITH A MECHANISM TWIST.
Vals best-in-series (easy 0.218, hard 0.347). Matched 2:0 = 39.1
(drop .25 / easy .69 / hard .433 — every bucket a d=1 record;
easy IMPROVED) — clears the pre-registered >36.6 threshold; paired
vs positional d=1: 50-31, McNemar p=0.045. BUT the ablation also
scores 39.1 (burst-on vs burst-off 8-8, p=1.0): the test-time
burst is causally INERT. Attribution: trajectory teacher-forcing
is a superior TRAINING SIGNAL for the merge adapter — the ten
transition regressions teach state-folding that pays off at every
visible-token carry step — not a working inference-time loop. The
program-long pattern holds: the loop at inference has never been
causal; the loop as training scaffold just produced the largest
adapter gain ever measured. ltf plateaued ~0.085 (from 0.116) —
the autonomous operator absorbs ~25% of the driven transitions.
Next-experiment implication (Nils to confirm): apply trajectory
TF to rung A itself (nothing deleted) — if the signal generalizes,
the 57.4 headline moves.
30. **The carried state as a metacognitive signal (pre-registered
2026-07-17 ~13:20, before running; Nils's design: "treat the carry
state as a metacognitive signal and build on it explicitly...
exploit what the loop provably does instead of fighting the
consumption wall. This never requires the frozen band to read
anything new").** Synthesis of three established results: the
carry's inference-time value is deferral/completion-state (430
divergence; drop McNemar p=0.0094); gate quality is worth ~9.6
points with CLASSIFIER quality binding on the pre-loop state
(item 20); the consumption wall blocks residual-injection reads
(items 25-28) but a NEW head can read anything, and its decisions
reach the frozen model as tokens. Design: answer-readiness head
g(s) — LayerNorm+MLP(64), class-balanced — on the carried L30
state at scratchpad line boundaries of rung-A rollouts. Labels
mechanical: fork a forced "Answer: " completion at each boundary,
check against gold. Harvests: train (427) and test (256) x
{carry, feedforward-control} — the FF arm trains the identical
head on the FF states. Readouts, all offline from one instrumented
pass (item-20 LUT methodology): (a) PRIMARY: carry-head test AUC
vs FF-head test AUC — if carry > FF, the recurrence carries
metacognitive signal the feedforward path lacks: the first
measured POSITIVE FUNCTIONAL ROLE for the carried state; (b)
theta sweep of "answer at first boundary with g>=theta else
natural end": accuracy vs mean scratchpad length vs the 57.4
fixed-format baseline; (c) ORACLE stop bound (best boundary per
item) = the ceiling adaptive stopping can reach. Predictions:
carry AUC > 0.65 and > FF AUC (the deferral evidence says the
signal exists); theta curve dominates fixed-format on tokens at
matched accuracy; oracle meaningfully above 57.4 (early-stop
rescues drift cases). Arm 2 (later): line-correctness /
deferral head gating re-expansion. Job: scripts/jobs/zzz_w_metacog.sh.
31. **Synthetic memory tokens: per-layer KV prefix (pre-registered
2026-07-17 ~15:50, before running; Nils's design: "the one read
mechanism a frozen transformer natively possesses is attention over
the KV cache... adapter maps L30 state -> KV entries at band
layers"; injection variant chosen by Nils: per-layer KV prefix).**
The read-side attack, composed with item 29: warm-start the arm-1
adapter (its burst COMPUTES — trajectory-trained, ltf 0.085 — but
is provably unread: ablation 8-8 p=1.0) and FREEZE it; train only
a KVMemoryAdapter (6.3M params: shared code-512 trunk + per-band-
layer k/v heads with per-layer geometry — gemma-4 mixes 256-d
sliding and 512-d global heads) mapping the 10 burst iterates to
post-RoPE (k,v) columns appended to every band layer's attention
during the answer scan and at generation. Engineering, validated
in smokes: the registered attention implementation is wrapped IN
PLACE (config name untouched -> mask construction identical;
disarmed = bit-exact, diff 0.0); zero-init value heads + learnable
per-layer gate bias (init -10) make the memory silent at init
(step-0 loss 0.108 = the arm-1 endpoint exactly); memory stays
armed through backward (checkpoint recompute). Output CE only,
200 steps, lr 1e-3. Eval: 0:0 sanity + 2:0 matched (burst +
memory); the ablation is item 29's matched cell BY CONSTRUCTION
(same frozen adapter and burst, memory absent) = 39.1. Decision:
>44.1 (>=+5 over 39.1) = the consumption wall was a READ-PATH
problem and native attention over state-derived KV breaches it;
within +-5 = even natively readable computed states go unused ->
the wall is not about the read mechanism either, and the hybrid/
metacog lines carry the program. In-flight note on item 29 arm 2
(tf+fr): matched 30.5 — the free-running term HURT (vs 39.1
tf-only), reinforcing the training-signal attribution: clean TF
transition gradients are the active ingredient.
Job: scripts/jobs/zzz_x_kvmem.sh.