903 lines
56 KiB
Markdown
903 lines
56 KiB
Markdown
# Pre-registered protocol: unified adapter eval (written before any test numbers)
|
||
|
||
Date: 2026-07-13, after val@499, before step-799 completion. No test-set
|
||
number for the unified adapter exists at time of writing.
|
||
|
||
1. **Primary loop depth: k=2, for both tasks.** Chosen on val CE with the
|
||
tie-breaking rule: prefer the SMALLEST k whose hard-cell val CE is within
|
||
0.01 nats of the best k. (Current val: MBPP hard k2−k4 = 0.004, GSM hard
|
||
k2−k4 = 0.006 → both ties → k=2.) The full k-curve is secondary/descriptive.
|
||
2. **Checkpoint selection criterion (scalar, fixed now):** mean of the two
|
||
hard-cell val CEs at k=2, tasks weighted equally:
|
||
crit = (gsm_hard_k2 + mbpp_hard_k2)/2. Lowest crit among saved checkpoints
|
||
wins. (At writing: step 499, crit = (0.475+0.205)/2 = 0.340.)
|
||
3. **Primary endpoints:** (a) MBPP test pass@1 hard-bucket at k=2 vs k=0;
|
||
(b) GSM8K test accuracy hard-bucket at k=2 vs k=0. McNemar, paired by item.
|
||
Overall accuracy is secondary (known to be underpowered at n=250/256).
|
||
4. **Same-harness rule:** all k, INCLUDING k=0 baselines, measured by the
|
||
prompt-only fast-path scripts (`generate_frozen_prompt`; k=0 = plain
|
||
cached generate inside the same function). No numbers carried over from
|
||
the full-position-loop harness.
|
||
5. **Known missing control (not covered by this run):** a same-size,
|
||
no-recurrence adapter (h -> h + MLP(h) at the L13->L14 boundary, no loop,
|
||
no band re-run) trained on identical data/objective. Until it exists,
|
||
"the loop does the work (vs. 1.6M new weights anywhere doing it)" is NOT
|
||
established. Queued as the next training run. Note the k=0 column is
|
||
gated off by construction and is a sanity check only — it is not this
|
||
control.
|
||
6. **Symmetric interference check (missing):** dedicated GSM8K prompt-only
|
||
adapter as the reference for "unified costs GSM nothing". Queued. Until
|
||
then the no-interference claim is one-directional (MBPP side only).
|
||
7. **Band-location ablation (pre-registered 2026-07-13, before any arm ran).**
|
||
Arms, all else identical (adapter size/init, data, curriculum, k, scripts;
|
||
MBPP): early L2-12, mid-narrow L17-27, late L24-34 (width-matched, 11
|
||
layers); shifted L6-22 (width-matched to the original 17). Reference:
|
||
workspace L14-30 (already run, 3 seeds). Prediction: workspace-centered
|
||
arms (L14-30, L17-27) exceed early/late on hard-bucket pass@1 at k=2-4 by
|
||
a wide margin; shifted intermediate. Falsification: near-parity across
|
||
arms demotes the lens claim from "locates where to loop" to "convenient
|
||
discovery tool"; to be reported either way. Primary readout: hard-bucket
|
||
pass@1 at k=4, e400 checkpoints throughout.
|
||
8. **Language commitments for the writeup:** the k0->k1 CE collapse (e.g.
|
||
4.36->0.18) is format/template learning expected from any trained adapter
|
||
and must not be quoted as evidence of routing/planning; informative
|
||
comparisons are within k>=1 cells only. Depth ordering k2 vs k4 deltas
|
||
(0.001-0.006 nats) are inside checkpoint jitter and must be described as
|
||
"k>=2 fits hard items equally well; k=1 slightly worse."
|
||
|
||
9. **Anchor/entrance sweep (pre-registered 2026-07-14 ~03:00, before any arm
|
||
ran).** Arms: bands (13,30), (12,30), (11,30) — injection point shifted
|
||
up from L14 at fixed tap L30; plus tap-23 = (14,23). All E2B/MBPP, same
|
||
recipe, e400, primary readout hard-bucket pass@1 at k=4. Competing
|
||
predictions: (a) "L14 special" (last full-attention KV-computing layer,
|
||
lens boundary) → anchor-13 drops; (b) "KV-channel count" (anchors 11-13
|
||
add 1-3 extra KV-recomputing attention channels) → holds or improves.
|
||
Body-length confound noted: earlier anchors lengthen the loop body; if
|
||
results shift, run matched-length control (12,28) before interpreting.
|
||
|
||
10. **L9 discriminator arm (pre-registered 2026-07-14 ~10:00, before running).**
|
||
Band (9,30): anchor at L9 — the only other full-attention, KV-computing
|
||
layer below the boundary — deep in the lens's sensor regime, tap fixed at
|
||
L30. Separates the two cliff explanations: (a) "lens boundary" predicts
|
||
catastrophic (like anchors 11-13: 25-34% overall); (b) "full-attention
|
||
KV-layer entry" predicts partial recovery (clearly above the L11-13
|
||
trend, i.e. >40% overall or hard >25%). Registered prediction: (a) —
|
||
the sensor-region content dominates; layer type does not rescue it.
|
||
Same recipe/checkpoint/eval as the anchor sweep (250 items, ks 0,2,4).
|
||
|
||
---
|
||
|
||
# Outcomes vs pre-registrations (scored 2026-07-14, after all arms completed)
|
||
|
||
1. **k=2 primary depth** — held. All primary comparisons reported at k=2;
|
||
k-curves descriptive. k≥2 plateau confirmed (k=8 gen-eval flat).
|
||
2. **Checkpoint criterion** — applied as written for the unified adapter.
|
||
Separately reported: val-CE is a poor proxy for generation accuracy;
|
||
later arms therefore pre-committed to fixed steps (e400) instead.
|
||
3. **Primary endpoints (unified adapter, k=2 vs k=0)** — (a) MBPP hard
|
||
3.6% → 28.6% (direction as predicted); (b) GSM hard 0% → 6.3%, overall
|
||
10.5% → 9.0% (no overall win — the math boundary result). Both reported.
|
||
4. **Same-harness rule** — held throughout (all final tables fast-path,
|
||
k=0 included).
|
||
5. **Missing weights control** — run: trained FF adapter = 17.9% hard,
|
||
exactly the untrained-loop level. Loop-vs-weights gap established.
|
||
6. **Symmetric interference check** — run (dedicated GSM adapter);
|
||
mixed-task training regressed both tasks; reported as negative result.
|
||
7. **Band-location ablation** — prediction CONFIRMED with a caveat:
|
||
L14-30 hard 43.6% ≫ early L2-12 (23.6%, overall destroyed 22.8%) and
|
||
shifted L6-22 (21.8%, overall 29.0%). Caveat discovered: L17-27 and
|
||
L24-34 are structurally null (KV sharing; k>0 ≡ k=0 bit-identical), so
|
||
the "mid-narrow beats late" half of the prediction was untestable at
|
||
E2B; the 12B replication (no shared KV) carries that weight instead.
|
||
8. **Language commitments** — honored in PAPER.md (k0→k1 CE collapse not
|
||
cited as planning evidence; k2-vs-k4 nats described as jitter).
|
||
9. **Anchor/entrance sweep** — prediction (a) "L14 special" CONFIRMED:
|
||
anchor-13 hard 17.9%/overall 34.4%; 12: 28.6%/30.8%; 11: 25.0%/25.2%;
|
||
monotone collapse below the boundary. Tap-23 arm died in training
|
||
(never rerun); exits 27/30/32/34 within seed noise, so exit choice is
|
||
free. Matched-length control not needed (results did not shift with
|
||
body length in the informative direction).
|
||
10. **L9 discriminator** — registered prediction (a) CONFIRMED: band
|
||
(9,30) overall 14.0-21.4%, hard ≤21.4% — catastrophic, like anchors
|
||
11-13, despite L9 being a full-attention KV-computing layer. The lens
|
||
boundary, not layer type, gates the retrofit.
|
||
|
||
11. **Recurrent-regime arm (pre-registered 2026-07-15, before training).**
|
||
Huginn-style retrofit on the frozen E2B band: RecurrentAdapter
|
||
(learned A,B init α·I/(1−α)·I + zero-init MLP), h0 = norm-scaled
|
||
noise, log-uniform random depth k∈[1,16], bptt=4, same data/steps/
|
||
checkpoint rule (e400 primary) as all merge arms. Eval ks 0,2,4,8,16,32
|
||
on the 250-item MBPP set. Competing predictions: (a) "amortization is
|
||
intrinsic to frozen-band retrofits" → performance plateaus by k≈4 at
|
||
or below the merge arm's level, no depth-monotone gain; (b) "fixed-
|
||
point behavior was an artifact of our fixed-shallow-k training"
|
||
(Huginn regime transfers) → monotone hard-bucket improvement past k=8
|
||
and reduced noise-seed sensitivity after training. Secondary readout:
|
||
path independence (two noise seeds → output agreement rate) at e400.
|
||
Known risk, stated in advance: 600 steps may be far too little for
|
||
this regime (McLeish et al. use ~50B tokens); a null here bounds the
|
||
cheap-retrofit budget only, not the regime.
|
||
|
||
12. **Parcae-constrained recurrent arm (pre-registered 2026-07-15, before
|
||
training; Prairie et al. 2026 parameterization).** Same as item 11 but
|
||
A = exp(−Δt·exp(a)) diagonal → ρ(A) < 1 by construction; init exactly
|
||
the α=0.3 merge (verified bit-equal at init). ρ(A) logged every 10
|
||
steps in BOTH arms. Theory-derived predictions, stated in advance:
|
||
(a) contraction ⇒ fixed point is a function of e ⇒ the Parcae arm
|
||
SATURATES in k (no depth-monotone gain) and its converged performance
|
||
is amortizable — if so, our deflationary result is a corollary of
|
||
ρ<1, and our observed k≈3–4 convergence is the geometric rate 0.3^k;
|
||
(b) the UNCONSTRAINED item-11 arm either drifts toward ρ≥1 (watch the
|
||
ρ log: divergent runs should show ρ≥1 before loss spikes) or, if it
|
||
gains monotone depth-performance, does so with ρ near 1 — the edge of
|
||
stability is where genuine iteration must live. Either outcome
|
||
formalizes "the anchor coefficient is the stability dial" as
|
||
"the anchor coefficient is the spectral radius".
|
||
|
||
13. **Per-depth adapter arm + free-ACT probe (pre-registered 2026-07-15,
|
||
before training).** (a) PerDepthAdapter: one merge adapter per
|
||
iteration (n=4, Bae-style depth-wise relaxation at the entrance;
|
||
breaks time-invariance — LTV, no fixed-point guarantee), standard
|
||
curriculum, e400, eval ks 0,2,4,8. Prediction: lands at or below the
|
||
distill/rung-2 amortization ceiling (~46% hard) because depth-indexed
|
||
weights add content, not state-evolution; exceeding it would show
|
||
per-iteration expressivity was binding and amend the deflationary
|
||
claim. Depths >4 reuse adapter 4 (stated: k=8 cell is then
|
||
fixed-point-like by construction). (b) Free-ACT probe on the standard
|
||
merge arm: record per-item convergence depth (cos>0.9995) at k=8 cap.
|
||
Predictions: accuracy unchanged vs fixed k (post-convergence no-ops);
|
||
mean k_conv ≈ 3; hard-labeled items converge SLOWER than easy ones
|
||
(adaptive compute allocates like ACT without any learned halting
|
||
parameter).
|
||
|
||
--- Outcome, item 11 (scored 2026-07-15, k=16/32 cells cancelled by
|
||
decision after k<=8): PREDICTION (a) SUBSTANTIALLY CONFIRMED, with one
|
||
twist. The unconstrained arm left contraction immediately (rho(A):
|
||
0.3 -> 3.4 by step 100, plateau ~4.5) yet trained smoothly — per-iteration
|
||
norm-matching converts magnitude explosion into directional churn, so
|
||
"rho>=1 => divergence" becomes "rho>=1 => divergence OR stationary churn"
|
||
under a norm projection. Consequences as predicted: substrate damage
|
||
(easy 98.4 -> ~69% at all k>0, far exceeding any contractive arm's tax),
|
||
val CE flat k=1..16 (stationary, not progressive), hard bucket at
|
||
merge level (35.7/39.3/42.9% at k=2/4/8 — a one-item-per-depth-doubling
|
||
crawl that at k=8 reaches what the contractive merge reaches at k=4,
|
||
never approaching the amortization ceiling from above). 4x parameters
|
||
bought nothing. Depth-monotone computation did not emerge at this budget.
|
||
|
||
14. **Tied-alpha arm (pre-registered 2026-07-15, before training).**
|
||
TiedAlphaAdapter: x = (1−a)⊙e + a⊙ŝ + MLP([e;ŝ]), a = σ(â) per-dim
|
||
learned, init a=0.3 everywhere (bit-equal to MergeAdapter at step 0,
|
||
verified). B tied to (1−a): convex combination keeps the LTI fixed
|
||
point on the e–ŝ segment (substrate-anchored by construction),
|
||
ρ = max(a) < 1 guaranteed, +d≈1.5K params. Standard curriculum,
|
||
s0 = band(e), e400, eval ks 0,2,4,8 on 250 items. This is the one
|
||
untested cell combining parcae's learnable decay with the merge's
|
||
anchoring. Predictions: (a) substrate fidelity preserved (easy ≈
|
||
merge's 88%, unlike both rec arms' ~70%) because anchoring, not
|
||
ρ, controls fidelity; (b) hard-bucket at merge level (no significant
|
||
gain — per-dim constant α is not where capability lives, per the
|
||
adaptive-α E2B result); (c) learned a drifts slightly DOWN from 0.3
|
||
(as in parcae). If (a) holds while rec arms failed it, the
|
||
fixed-point-location dial is causally isolated: same learnable-decay
|
||
freedom, only the tie to (1−a) differs from parcae.
|
||
|
||
--- Outcome, item 12 (scored 2026-07-15): prediction (a) CONFIRMED in its
|
||
dynamics half, REFUTED in its fidelity half — and the refutation is the
|
||
finding. Dynamics: rho stayed in (0,1) throughout (0.300 -> 0.292, the
|
||
optimizer drifting MORE contractive when confined to the stable region);
|
||
loss trajectory as good as or better than the unconstrained arm at every
|
||
checkpoint (the rec arm's flight to rho~4.5 was epiphenomenal — all fit
|
||
lives in the MLP); eval saturates completely (hard 42.9/42.9/39.3/39.3/
|
||
39.3 at k=2/4/8/16/32, easy flat ~71%). Fidelity: easy items were NOT
|
||
preserved (71% vs the merge's 88.5%) despite guaranteed contraction —
|
||
substrate fidelity is controlled by fixed-point LOCATION (anchored B +
|
||
curriculum), not by rho. Conclusion: stability and fidelity are
|
||
independent dials (fig_phase.png); the Parcae constraint delivers exactly
|
||
what it promises (robust training, convergence, certified tail gradients)
|
||
and exactly nothing more. Item 14 (tied-alpha) is the causal isolation of
|
||
the fidelity dial.
|
||
|
||
15. **Fidelity factorial + capacity control + seed (pre-registered
|
||
2026-07-15 ~03:15, before any of these arms ran; overnight batch).**
|
||
The fidelity loss of both rec arms (easy 88.5 -> ~71%) confounds three
|
||
deltas from the winning merge: (i) learned B, (ii) random-depth
|
||
training instead of the difficulty->depth curriculum, (iii) noise s0.
|
||
Item 14 (tied-alpha) tests (i) with anchoring. New single-variable
|
||
cells, everything else = standard merge recipe (fixed B, band(e) s0,
|
||
curriculum, e400, eval ks 0,2,4,8 on 250 items):
|
||
a. merge+randk — only (ii) changed (log-uniform k in [1,16], bptt 4).
|
||
b. merge+noises0 — only (iii) changed.
|
||
c. merge h=2048 — capacity control for the per-depth arm (6.4M
|
||
shared vs 6.4M depth-indexed): if per-depth beats the ceiling
|
||
but h2048 does not, time-variation (not capacity) is credited;
|
||
if both do, it was capacity all along.
|
||
d. parcae seed 1 — robustness of the fidelity refutation.
|
||
Predictions: (a) and (b) each cost a few points of easy at most
|
||
(anchored fixed point dominates); neither reproduces the ~17-point
|
||
drop — the culprit is the learned/free B (with item 14 as the
|
||
positive control). h2048 stays at the ceiling (hard <=46%), fidelity
|
||
intact. parcae s1 reproduces easy ~71% within seed noise.
|
||
|
||
--- Outcome, item 13a (scored 2026-07-15): prediction CONFIRMED — per-depth
|
||
lands below/at the ceiling, never above. Detail is instructive: fidelity
|
||
preserved throughout (easy 88.5/89.3/86.9 at k=2/4/8 — anchored B), but
|
||
hard-bucket content is DEPTH-STRANDED: 17.9% at k=2 (adapters 3-4, which
|
||
hold the hard-trained content, never execute), 35.7% at k=4, 42.9% at k=8
|
||
— where depths 5-8 reuse adapter 4, i.e. the architecture reverts to
|
||
shared-map iteration and the fixed-point mechanism collects the remaining
|
||
gain. Time-variation adds a fragility (content unavailable except at its
|
||
training depth) and no capability; map-sharing is load-bearing for the
|
||
anytime-usable gain. Depth-4 adapter overfit visible in val (hard k4 CE
|
||
0.188@99 -> 0.371@599) — LTV concentrates small-pool overfitting into
|
||
single depths.
|
||
|
||
--- Outcome, item 13b (scored 2026-07-15): accuracy prediction CONFIRMED
|
||
(k=8 halt run 52.0/90.2/42.9 = plateau level); convergence predictions
|
||
REFUTED. Per-item state-cosine (thresh 0.9995, k=8 cap): k_conv
|
||
distribution 4:3, 5:57, 6:47, 7:17, never-within-8:126 — mean ~7, and NO
|
||
difficulty gradient (easy 7.01 vs hard 7.00). The earlier "bit-exact by
|
||
k~3-4" was the single dynamics-probe example, not the population: outputs
|
||
plateau by k~2-4 while the state keeps drifting at 1e-3..1e-4 cosine
|
||
scale; the fixed point is an OUTPUT-stable orbit (suffix layers + decode
|
||
wash out residual state motion), not a literal state fixed point for most
|
||
prompts. Free-ACT via state-cosine therefore yields no early exit at this
|
||
threshold, and no ACT-like difficulty allocation falls out for free —
|
||
output-level halting signals would be needed. Paper's dynamics claims
|
||
softened accordingly.
|
||
|
||
--- Outcome, item 14 (scored 2026-07-15): ALL THREE PREDICTIONS CONFIRMED.
|
||
(a) Fidelity fully preserved: easy 93.4/91.0/90.2 at k=2/4/8 (merge:
|
||
92.6/88.5; parcae with identical decay freedom but untied B: ~71%) —
|
||
the free B is causally isolated as the fidelity culprit, the anchoring
|
||
tie as the protection. (b) Hard at merge level exactly (35.7/42.9/39.3 =
|
||
merge's k-curve within noise); no gain from the freedom. (c) Learned a
|
||
essentially unmoved: mean 0.298, range [0.285, 0.310], 0/1536 dims moved
|
||
>0.05 from init — the anchor coefficient is not a useful learnable DOF;
|
||
hand-tuned 0.3 was already optimal. Recipe consequence: fixed-alpha
|
||
anchored merge is the recommended design; learnable-alpha safe but
|
||
pointless, learnable-B harmful, per-depth strands the gain.
|
||
|
||
16. **Code→GSM8K cross-task transfer (pre-registered 2026-07-15 ~14:10,
|
||
before running).** The MBPP-trained loop adapter (adapter_code, s0) and
|
||
the noise-s0 variant evaluated on GSM8K test (n=256, prompt-only loop,
|
||
same harness as eval_gsmonly). Extends the transfer-distance ladder
|
||
(HumanEval tie -> LCB trained-hurts) across tasks. Predictions:
|
||
(a) hard-bucket gain ~0 (plan content is task-local; GSM8K needs
|
||
evolving state, not static plans); (b) easy items damaged at k>0
|
||
(~93 -> 50-70%), comparable to or worse than the GSM-trained merge —
|
||
substrate damage on GSM8K is perturbation-driven and content-agnostic;
|
||
(c) overall at k>0 below k=0 (no rescue). If instead hard gains
|
||
appear (>5 points), plan-shaped content is partially task-general —
|
||
would weaken the task-local claim from LCB.
|
||
|
||
--- Amendment to item 15 (2026-07-15 ~13:15): noise-s0 arm EXCEEDED
|
||
prediction (b) upward: hard 50.0/53.6/50.0 at k=2/4/8 with easy 88-90%
|
||
— nominally the best hard cells of the project (merge best 46.4; seed
|
||
mean 37.5±5.5). Paired vs tied-alpha (only same-day per-item baseline):
|
||
discordants 5-1/3-0/3-0 in noise-s0's favor, each k p≈0.22-0.25 at n=28
|
||
— consistent direction, not individually significant. Denoising
|
||
interpretation: training the loop to reach the fixed point from noise
|
||
regularizes the content. SEED ARMS QUEUED (s1, s2, same recipe/eval,
|
||
pre-registered here): if seed-mean hard(k=4) > 46.4 (the merge's best
|
||
single cell), the recommended recipe gains noise-s0; if seed mean falls
|
||
back into 37-46, it was a lucky seed.
|
||
|
||
--- Outcome, item 15c (h2048 capacity control, scored 2026-07-15): the
|
||
per-depth exoneration is CLEAN — shared 6.4M params reach hard 42.9/53.6/
|
||
50.0 at k=2/4/8 vs per-depth's 17.9/35.7/42.9 at the same capacity;
|
||
time-variation is strictly worse than weight-sharing at matched params.
|
||
Fidelity prediction confirmed and exceeded (easy 95.1% at k=2 — best
|
||
looped fidelity of the project; 90.2% at k=4/8). Ceiling prediction
|
||
(hard <= 46%) REFUTED UPWARD like noise-s0: k=4/8 at 53.6/50.0. Two
|
||
independent variations (noise s0, 4x MLP) now sit at 50-54% where the
|
||
original merge reached 46.4 — suggests 46.4 was an UNDER-estimate of the
|
||
recipe family's level, not a ceiling it defined. The distill-parity
|
||
deflation claim is unaffected statistically (53.6 vs 45.7 at hard n=28
|
||
is within noise) but the language "every regime tops out at the same
|
||
ceiling" should become "at the same level within noise" — pending the
|
||
noise-s0 seed arms.
|
||
|
||
--- Outcome, item 15d (parcae seed 1, scored 2026-07-15): CONFIRMED —
|
||
the fidelity refutation replicates. easy 70.5/73.0/72.1 at k=2/4/8
|
||
(seed 0: 72.1/71.3/70.5); hard 32.1/39.3/35.7 (seed 0: 42.9/42.9/39.3,
|
||
ordinary seed spread at n=28). Two-seed conclusion: contraction-with-
|
||
free-B loses ~17 points of easy items regardless of seed; the phase
|
||
diagram's Parcae point is solid.
|
||
|
||
--- Outcome, item 16 (code->GSM8K transfer, scored 2026-07-15): ALL THREE
|
||
PREDICTIONS CONFIRMED, emphatically. MBPP-trained loop on GSM8K: hard
|
||
0.8-1.6% at every k (prediction a: ~0 gain — plan content is task-local);
|
||
easy 93.1 -> 27.6-44.8% (prediction b: damaged, in fact WORSE than the
|
||
GSM-trained merge's 48%); overall strictly below k=0 at every k>0
|
||
(prediction c). noise-s0 variant identical (easy 34.5, hard 1.6). The
|
||
transfer-distance ladder ends cleanly: near (HumanEval) tie, far-code
|
||
(LCB) trained-hurts, cross-task (GSM8K) trained-content actively toxic
|
||
while gaining nothing. Task-locality of the learned content is now a
|
||
three-point monotone result.
|
||
|
||
--- Closure of the item-15b/15c "ceiling nudged upward" question
|
||
(2026-07-15, after ns seeds): LUCKY SEED, per the pre-registered rule.
|
||
noise-s0 hard(k=4) across seeds: 53.6 / 39.3 / 35.7 -> seed mean 42.9,
|
||
inside the 37-46 band. Fidelity across seeds intact (easy 90.2-94.3 —
|
||
the factorial conclusion is seed-robust); the 50-54% cells (ns seed 0,
|
||
h2048 single seed) were upper-tail draws of the same distribution the
|
||
merge's 46.4 came from. No recipe amendment; the abstract's original
|
||
"same level within noise" framing stands; single-cell records are not
|
||
levels — only seed means are.
|
||
|
||
17. **GSM-only, current recipe (pre-registered 2026-07-15 ~20:45, before
|
||
running).** train_merge_unified.py --tasks gsm: MergeAdapter, prompt-
|
||
only loop, curriculum, GSM8K data ONLY — removes the mixed-task
|
||
interference confound from the adapter_uni run, completing the
|
||
"winning recipe trained on GSM" question. Eval: prompt-only, n=256,
|
||
ks 0,1,2,4, e400. Predictions: (a) hard <= 10% at every k (supervision
|
||
density is structural: ~3 answer tokens; the recipe's dense-output
|
||
ingredient cannot exist here); (b) easy damaged at k>0 (to 40-70%);
|
||
(c) overall never beats k=0. If hard exceeds 15% or overall beats
|
||
k=0, task interference in the mixed run was masking a real GSM
|
||
capability — would reopen the GSM chapter.
|
||
|
||
Scope note (item 17): the design-space arms of items 11-15 are NOT
|
||
crossed with GSM8K, deliberately. Exclusion by dominance: fidelity-
|
||
failing regimes (rec, parcae) cannot improve on a task MORE fidelity-
|
||
fragile than MBPP; architecture-failing (per-depth) and equivalent
|
||
(tied-alpha -> merge) and k-placement-only (randk) and same-family
|
||
(noise-s0, h2048) variants have no mechanism by which task change
|
||
could invert their MBPP verdict. Only the recipe family's best member
|
||
(this item) is informative on GSM8K.
|
||
|
||
--- Outcome, item 17 (GSM-only, current recipe, scored 2026-07-15):
|
||
predictions (a) and (b) CONFIRMED, (c) nominally exceeded but not
|
||
meaningfully. hard 8.7/5.5/4.7% at k=1/2/4 (below the 10% bar; nowhere
|
||
near the 15% reopen threshold); easy 93.1 -> 48-52% at k>0; overall
|
||
11.7/10.9/10.2 vs k0's 10.5 — the k=1 cell is +1.2 points nominal
|
||
(~3 items at n=256, not significant), the rest below. Removing the
|
||
mixed-task interference bought ~2 points over adapter_uni (9.4 -> 11.7
|
||
at k=1) — interference was real but marginal, not masking a capability.
|
||
The GSM8K chapter is closed: the recipe family's best member, trained
|
||
on GSM alone in the correct regime, delivers no usable gain and the
|
||
standard fidelity damage; combined with the scope note, the boundary
|
||
claim (structural: supervision density + state-evolution bottleneck)
|
||
is fully supported.
|
||
|
||
18. **E1: learned per-prompt halting gate (pre-registered 2026-07-16
|
||
~00:20, before any arm runs; PLAN_SELFPACED.md).** HaltingMergeAdapter:
|
||
frozen-recipe merge + ACT-style halting head on the last prompt
|
||
position's workspace state; soft state-mixture training, CE + lambda *
|
||
E[iters], penalty warmup at step 100; NO difficulty curriculum (mixed
|
||
batches — the gate must discover the allocation). k_max=4, e400/e600
|
||
checkpoints, deploy = sequential halting at 0.5 cumulative mass,
|
||
generation via frozen-prompt at per-item k*. Arms: lambda in
|
||
{0, 1e-3, 1e-2}, seed 0. Eval: 250 items, vs anchors k=0 (0.488),
|
||
uniform merge k=4 (0.512/0.885/0.464), probe-gate E0 (0.520/0.975/0.286).
|
||
Predictions: (a) some lambda gives overall >= 0.512 at mean E[k] <=
|
||
2.4 (60% of uniform-4); (b) easy >= 0.95 at that lambda; (c) k*-vs-hard
|
||
point-biserial r > 0.3; (d) hard >= 0.286 (beats E0's frozen probe).
|
||
Collapse (E[k] pinned at 1 or 4 for all lambda) falsifies E1 and
|
||
triggers the plan's kill criterion. lambda=0 control isolates whether
|
||
the CE gradient alone moves the gate (expected: barely — penalty
|
||
provides the pressure).
|
||
|
||
Item 18 amendment (2026-07-16 ~23:45, before results): arms run on a
|
||
rented 4xH100 node in parallel instead of the Spark queue; a fourth
|
||
arm (lambda=1e-3, seed 1) is added for immediate seed replication of
|
||
the expected-winner penalty. Spark's queued gate jobs will be dropped
|
||
to avoid duplication. Everything else per registration.
|
||
|
||
--- Outcome, item 18 (scored 2026-07-16 ~00:40): predictions (b), (c)
|
||
REFUTED, (a) marginal miss, (d) trivial pass. All arms converge to
|
||
UNIFORM depth (lambda 0/1e-3/1e-2 -> E[k] 4/2-or-4/1; the two 1e-3 seeds
|
||
picked different plateaus — degenerate penalty landscape), r = 0.000
|
||
everywhere. Mechanism identified and consistent with prior findings:
|
||
teacher-forced CE is depth-flat (stationarity), so CE provides no
|
||
per-item depth gradient; the penalty alone cannot teach selectivity.
|
||
The state DOES carry the signal (E0 probe: train acc 1.0) — the failure
|
||
is the training signal, not the representation. E1-as-designed is dead;
|
||
kill criterion NOT fully triggered (E2 untested, and the mechanism
|
||
points at a repair).
|
||
|
||
19. **E1b: label-supervised halting head (pre-registered 2026-07-16
|
||
~00:45, before running).** Freeze the curriculum merge (adapter_code
|
||
s0); train ONLY the halting head (BCE): target halt=0 at iterations
|
||
below the label's depth (easy->1, hard->4, per STaR label), halt=1 at
|
||
or above it. 300 steps, mixed batches, head-only params. Eval: gated
|
||
eval as item 18, n=250. Predictions: (a) r(k*, hard) > 0.5 (the head
|
||
is a trained difficulty classifier now); (b) easy >= 95% at k*=1
|
||
(near-E0's 97.5); (c) hard >= 35.7% (>= best uniform arm, via better
|
||
recall than E0's frozen probe: more than 18/28 hard items routed
|
||
deep); (d) overall >= 52.0 at E[k] <= 2.2. If (c) fails while (a,b)
|
||
hold, halting-head recall saturates at probe level and gate quality,
|
||
not gate training, is the binding constraint.
|
||
|
||
--- Outcome, item 19 / E1b (scored 2026-07-16 ~01:15): prediction (c)
|
||
CONFIRMED (hard 39.3 >= 35.7 at mean k* 2.18), (a) FAILED at r=0.217
|
||
(selectivity real — hard routed 2x deeper than easy (2.18 vs 1.08), the
|
||
program's first nonzero gate correlation — but weak at deploy), (b,d)
|
||
FAILED for a traced design reason: halted_k_per_item lacked k*=0, so easy
|
||
items were forced through >=1 iteration and landed on the merge's WORST
|
||
easy depth (k=1: 85.2%); E0's 97.5% came precisely from k=0 routing.
|
||
E1c amendment (pre-registered before running, same session): pre-loop
|
||
halt consult on s_0 enabling k*=0; targets easy->0, hard->4; threshold
|
||
0.5 unchanged (calibration deferred unless E1c misses). Predictions:
|
||
easy >= 95%, hard >= 35.7%, r >= 0.4, overall >= 51.2 at E[k] <= 1.5.
|
||
|
||
--- Outcome, item 19 / E1c (scored 2026-07-16 ~01:50, Spark re-run):
|
||
prediction (b) CONFIRMED — easy 95.9% with mean k*=0.11 (the k=0 routing
|
||
fix worked perfectly for easy items); (a) FAILED (r=0.220, unchanged);
|
||
(c) FAILED HARD — hard 21.4% at mean k*=1.5: with the pre-loop halt
|
||
consult, weakly-discriminated hard items now exit at k*=0/1 where before
|
||
they got >=1 iteration; the recall problem became more expensive, not
|
||
better. (d) FAILED (overall 50.0 at E[k]=0.74). Net Pareto: E1c =
|
||
(50.0 overall, 95.9 easy, 21.4 hard, 0.74 mean iters — 82% compute
|
||
saved); E0 probe-gate = (52.0, 97.5, 28.6, ~2.2) still dominates on
|
||
accuracy. The learned-head line has ONE identified un-tried knob:
|
||
deploy-threshold calibration on val for hard-recall (the 0.5 threshold
|
||
is arbitrary; raising it routes more items deep, trading easy tax for
|
||
hard recall — a tunable curve E0 cannot offer). E1 arc summary for
|
||
PLAN_SELFPACED: gating machinery works end-to-end, easy-item protection
|
||
and compute savings are demonstrated and cheap; difficulty-selective
|
||
DEPTH allocation remains unsolved at 3K-param-head scale — binding
|
||
constraint is classifier quality on the k=0/s0 state, exactly where E0
|
||
started. Next per plan: threshold sweep (cheap) before any E2.
|
||
|
||
20. **E1 threshold curve + oracle bound (pre-registered 2026-07-16 ~02:15,
|
||
before running).** Phase 1: record E1c head's halt probabilities per
|
||
test item (one GPU pass). Phase 2: per-item outcomes for the frozen
|
||
curriculum merge at k=0/1/2/4 (four generation sweeps, tag merge_lut —
|
||
doubles as the reusable gate-evaluation lookup table and supplies the
|
||
long-missing per-item logs for the canonical merge). Phase 3 (offline):
|
||
gated accuracy at thresholds .3-.99 by composing k*(theta) with the
|
||
lookup; plus the ORACLE gate (best k per item) = the ceiling any gate
|
||
can reach with this merge. Predictions: (a) some theta gives hard >=
|
||
32% with easy >= 93% and E[k] <= 2.2 (dominating E0 on compute at
|
||
comparable accuracy); (b) the curve is monotone in theta; (c) oracle
|
||
overall >= 55% — if so, gate-quality headroom is large and further
|
||
gate work is justified; if oracle < 53%, gating this merge is nearly
|
||
saturated and the program pivots to E2 or closes.
|
||
|
||
--- Outcome, item 20 (scored 2026-07-16 ~02:35): (b) CONFIRMED — clean
|
||
monotone threshold curve (hard 7->50%, easy 96.7->87.7%, E[k] 0.43->2.63
|
||
across theta .3->.99). (a) FAILED — no theta reaches easy>=93 AND
|
||
hard>=32; at matched easy the E0 frozen probe dominates the entire
|
||
learned-head curve: the BCE-trained 3K head is strictly worse than the
|
||
class-balanced logistic probe it was meant to replace. (c) CONFIRMED,
|
||
emphatically: ORACLE gate = 59.6 overall / easy 100% / hard 64.3% at
|
||
E[k]=0.24. Key insight: hard items are DEPTH-DIVERSE — 18/28 solvable at
|
||
some k in {0,1,2,4} but no single k solves more than 13; a third of the
|
||
hard bucket lives in per-item depth selection. Program continues per
|
||
rule; binding constraint quantified: gate quality is worth ~9.6 overall
|
||
points (50.0 deployed vs 59.6 oracle). Also noted: the LUT re-run of the
|
||
canonical merge shows small systematic drift vs the Jul-13 eval (k4 hard
|
||
46.4 identical, k1/k2 hard 3 items lower) — the LUT (per-item, single
|
||
harness run) is now the canonical reference. Next candidates, in cost
|
||
order: (i) deploy E0's probe AS the gate against the LUT (free,
|
||
offline); (ii) stronger classifier (multi-position features, more data,
|
||
calibrated threshold); (iii) oracle-gap error analysis on the hard items
|
||
no fixed k solves but some k does.
|
||
|
||
21. **E2 stage A: dense short-CoT supervision through the carry
|
||
whiteboard, GSM8K (pre-registered 2026-07-16 ~02:55, before running;
|
||
PLAN_SELFPACED E2 / the hybrid from the internalization discussion).**
|
||
Prep: harvest TERSE verified CoTs ("at most 3 short steps", answer-
|
||
verified, STaR filter) for GSM train. Arms: (A) carry regime
|
||
(k=2 prefill, pauses easy p=2 / hard p=6) trained with CE on
|
||
scratchpad+answer (~30-60 dense tokens — the ingredient every latent
|
||
GSM arm lacked); (B) CONTROL: identical supervision, feedforward
|
||
adapter, no recurrence. Eval: GSM test 256, grid 0:0 (base), 2:2,
|
||
2:6; e400 checkpoints. Predictions: (a) arm A beats every previous
|
||
GSM arm's overall (>12.1%) — dense supervision is the binding fix;
|
||
(b) the A-vs-B delta isolates the whiteboard: if A > B by >=3 points
|
||
overall, recurrence adds value beyond visible-scratchpad training;
|
||
if A ~= B, the scratchpad text alone carries it (deflation, GSM
|
||
edition); (c) easy-bucket damage smaller than answer-only carry's
|
||
(83->45%) because training and deployment output formats now match.
|
||
Honest note: arm outputs are VISIBLE tokens (~40) — this is the
|
||
budget-CoT-with-loop hybrid, a scope change from latent planning,
|
||
run at Nils's explicit direction ("do gsm8k and such").
|
||
|
||
--- Outcome, item 21 (scored 2026-07-16 ~06:30). Harvest: 427 verified
|
||
terse CoTs (292 hard, 135 easy; 73.5% yield, ~3 min). Grid (n=256,
|
||
base 10.9/93.1/0.8/0.0 overall/easy/hard/drop):
|
||
arm A carry 2:2 53.9 (82.8 / 60.6 / 37.0)
|
||
arm A carry 2:6 57.4 (72.4 / 63.8 / 45.0)
|
||
control FF 2:2 49.6 (79.3 / 59.8 / 28.0)
|
||
control FF 2:6 54.7 (82.8 / 66.9 / 31.0)
|
||
(a) CONFIRMED, dramatically: best cell 57.4% vs the previous best GSM
|
||
arm's 12.1% — a 5x jump to full-CoT territory (~53%) at ~1/4 the
|
||
visible tokens. Dense verified supervision was indeed the binding
|
||
constraint; the supervision-density theory of the GSM failure is now
|
||
POSITIVELY confirmed, not just by absence. (b) MIXED: A-vs-B delta
|
||
+4.3 at p=2 (clears the >=3 bar), +2.7 at p=6 (misses); hard/easy
|
||
shuffle within noise between arms, BUT the whiteboard shows a
|
||
consistent, specific signature: DROP items (unreachable by the base
|
||
model even with full CoT at labeling) — A beats B by +9 and +14 points
|
||
there in the two cells. Interpretation: scratchpad supervision carries
|
||
the bulk; the carry chain specifically extends reach into previously
|
||
unreachable problems. McNemar (scored 2026-07-16 morning): overall
|
||
A-vs-B not significant in either cell (2:2 discordants 34-23 p=0.185;
|
||
2:6 33-26 p=0.435), but the pre-identified drop-bucket signature IS:
|
||
2:6 drop discordants 20-6, exact McNemar p=0.0094 (survives
|
||
Bonferroni x4 = 0.038); 2:2 drop 16-7, p=0.093 (same direction,
|
||
marginal). Verdict: the whiteboard's edge is real and specific to
|
||
extending reach into drop items, not a general lift over matched
|
||
supervision. (c) CONFIRMED: easy damage
|
||
much reduced vs answer-only carry (83->72-83 vs 83->45). Ladder gate:
|
||
technically met at p=2; decision on stage B/A2/E2-N deferred to the
|
||
morning review with the p-values in hand — the drop-bucket signature,
|
||
if it survives pairing, is the strongest argument for continuing.
|
||
|
||
22. **E2-L rung B: internalization ladder, front-first step deletion
|
||
(pre-registered 2026-07-16 ~10:40, before running; Nils chose stage B
|
||
over A2/E2-N at the morning review. Gate state: pre-registered +3
|
||
overall met on points (+4.3 at p=2); paired McNemar overall ns, but
|
||
the drop-bucket signature significant, 2:6 p=0.0094).** Design:
|
||
delete the first d scratchpad lines of each verified terse CoT
|
||
(d=1,2,3 — front-first: the deleted computation must ride the
|
||
pause-chain before the visible remainder), each deleted step replaced
|
||
by 10 pauses (median step = 10 tokens, compute-matched); unparseable
|
||
cots (14/427 without exactly one Answer line) pass through intact.
|
||
Step-count distribution 1/2/3/4/5+: 11/164/196/27/15 — so d=3 is
|
||
effectively rung C (pauses only) for ~87% of items. Each rung
|
||
warm-starts from the previous (d=1 from rung-A e400), brief retrain:
|
||
200 steps, LR 3e-4 cosine, seed 0. Eval: GSM test n=256, cells 0:0
|
||
(k=0 sanity, expect ~base 10.9), 2:(2+10d), 2:(6+10d); e200
|
||
checkpoints; per-item logs kept so rung-vs-rung McNemar is offline.
|
||
Known approximation, stated in advance: items with fewer than d
|
||
steps train at smaller effective p than the eval cell
|
||
(ndel=min(d,n_steps)). Predictions: (a) d=1 best cell within 5
|
||
points of rung A's 57.4 — one step fits the recurrence budget (the
|
||
drop-bucket reach evidence says the whiteboard already carries
|
||
step-sized computation); (b) monotone decline across d; (c) at d=3
|
||
accuracy stays above BOTH base (10.9) and cold answer-only carry
|
||
(9.4) — curriculum beats cold training even where the ladder breaks.
|
||
Deliverable: the break rung = first d whose best cell falls >=5
|
||
points below the previous rung's best — the measured capacity of
|
||
this recurrence budget to absorb computation. Job:
|
||
scripts/jobs/zzz_m_gsm_rungb.sh (single submit, ~3x(40min train +
|
||
eval) on the Spark).
|
||
|
||
--- Outcome, item 22 (scored 2026-07-16 ~15:30). k=0 sanity row
|
||
reproduced base exactly in all three rung evals (10.9/93.1/0.8/0.0).
|
||
Ladder (best cell overall, n=256): rung A 57.4 -> d=1 31.6 (2:12) ->
|
||
d=2 18.4 (2:26) -> d=3 19.1 (2:32). (a) FAILED, decisively: the break
|
||
rung is d=1 — deleting ONE compute-matched step costs 25.8 points
|
||
(easy 83->62, hard 64->36, drop 45->17); the recurrence cannot absorb
|
||
even one step's computation at this budget/recipe. (b) monotone
|
||
through d=2, then a plateau (d=2 18.4 -> d=3 19.1, within n=256
|
||
noise): the decay bottoms out ABOVE the floor rather than collapsing
|
||
to it. (c) CONFIRMED: d=3 (pause-only for ~87% of items) = 19.1 vs
|
||
base 10.9 — paired McNemar 33-12 discordants, p=0.0025 — and vs cold
|
||
answer-only carry 9.4: the curriculum-reached latent loop DOUBLES the
|
||
cold-trained equivalent. Composition of the d=3 edge: easy is
|
||
DAMAGED (58.6 vs 93.1 base) while hard (18.9 vs 0.8) and drop (8.0
|
||
vs 0.0) are lifted — the latent loop trades easy-bucket reliability
|
||
for reach, echoing item 21's drop-bucket signature in latent form.
|
||
Deliverable: measured capacity of this recurrence budget = a plateau
|
||
at ~19% overall / ~2x the cold floor; the visible scratchpad carried
|
||
the other ~38 points. Val-loss note: hard val rose with d (0.36 ->
|
||
0.56 -> 0.59) while easy fell (d=3 0.13, mostly answer-line targets)
|
||
— CE fit does not track eval accuracy at deep rungs. Interpretation
|
||
for the plan: rung C' (no pauses) is moot as a capability claim — C
|
||
already plateaued at 19; the E2-L line's value is now the 2x-over-cold
|
||
curriculum effect + the reach trade, not scratchpad-free parity. Next
|
||
knobs if the line continues: longer per-rung retraining, finer
|
||
deletion schedule (fractional rungs), or E2-N noise-hardening stacked
|
||
on the rungs.
|
||
|
||
23. **E2-L d=1 capacity-ceiling controls (pre-registered 2026-07-16
|
||
~15:55, before running; Nils: "run that one control").** Item 22's
|
||
break-at-d=1 has two untested confounds; one arm each, single-knob
|
||
changes from item 22's d=1 (both warm-start rung-A e400, seed 0):
|
||
arm A "x600" = 600 steps instead of 200 (3x training; tests the
|
||
brief-retrain objection — note d=1 val had plateaued at 0.36, so
|
||
prediction is NO recovery); arm B "pp30" = 30 pauses per deleted
|
||
step instead of 10 (3x latent bandwidth, same 200 steps; tests
|
||
whether inert-pause capacity, not training, binds). Eval n=256:
|
||
arm A cells 0:0, 2:12, 2:16 (e600); arm B cells 0:0, 2:32, 2:36
|
||
(e200). Decision rule, stated in advance: item 22 d=1 best = 31.6;
|
||
a control within +-5 points confirms the ceiling on that axis;
|
||
>=+5 (>36.6) reopens the ladder on that knob (longer schedules or
|
||
wider pause-chains); if BOTH land within +-5, the d=1 ceiling is
|
||
confirmed structural and the ladder chapter closes as scored.
|
||
Job: scripts/jobs/zzz_n_rungb_ctrl.sh.
|
||
AMENDMENT (2026-07-16 ~19:05, mid-run, Nils's call): arm A's eval
|
||
skipped to save ~50min GPU — the verdict was already decided by
|
||
training evidence (train loss memorized to 0.02-0.10 by step 200;
|
||
val easy 0.386->0.397->0.426, hard 0.357->0.376->0.421 across
|
||
e200/e400/e600 — monotone UP, textbook overfit, no recovery; k=0
|
||
sanity row reproduced base before the kill). Arm A scored from the
|
||
val trajectory: training time is NOT the binding constraint. Arm B
|
||
(pp30) runs in full as registered.
|
||
|
||
--- Outcome, item 23 (scored 2026-07-16 ~23:55; rc=0). k=0 sanity rows
|
||
reproduced base in both arms' evals (arm A's before its eval was
|
||
skipped). Arm A (x600): scored from val trajectory per amendment — no
|
||
recovery, mild overfit; training-time axis CONFIRMS the ceiling. Arm B
|
||
(pp30): best cell 29.3 (2:36; 28.5 at 2:32) vs item-22 d=1's 31.6 —
|
||
WITHIN the +-5 band, slightly below: 3x latent positions bought
|
||
nothing (hard 33.9/36.2 vs 36.2; drop 14 vs 17; easy 51.7-55.2 vs
|
||
62.1). Decision rule: BOTH axes confirm -> the d=1 break is
|
||
STRUCTURAL. Neither longer training nor a longer pause-chain lets
|
||
this carry absorb one scratchpad step; the binding constraint is the
|
||
carried state itself (its per-position expressivity/fidelity, not its
|
||
compute budget). Converging evidence from the same evening's
|
||
microscopy (probe_discount*/probe_gsm*): the board natively carries
|
||
plans ("Multiply", "subtract"), coarse magnitudes ("sixty/eighty"),
|
||
and completion-state ("plus" kept alive at the 430 divergence — carry
|
||
defers where FF commits a wrong digit at 99.3%), while exact digits
|
||
appear only just-in-time, 1-2 positions pre-emission. The ladder
|
||
chapter closes as scored in item 22; the loop program's live paths
|
||
are state-side (rung-2 band-LoRA / wider merge, E2-N hardening,
|
||
coarse-target auxiliary supervision matched to the medium) or the
|
||
hybrid (A2), not longer/denser pause-chains.
|
||
|
||
24. **E2-L d=1 with a trainable band: loop-only band-LoRA
|
||
(pre-registered 2026-07-17 ~00:30, before running; Nils: "can we
|
||
unfreeze the entire band and try this again?").** Item 23 located
|
||
the constraint in the carried state's per-position transformation;
|
||
this is the state-side attack. Full unfreeze rejected in design
|
||
(642M params vs 427 examples, and it would break the frozen-model
|
||
guarantee); instead LoopLoRA (lora_band.py, built for the rung-2
|
||
design): rank-16 deltas on q/v/down of EVERY band layer (L14-30,
|
||
uniform scale 1.0, 4.8M params), active ONLY during band re-runs —
|
||
initial forward and k=0 stay bit-exact by construction. Otherwise
|
||
identical to item-22 d=1: front-first deletion, 10 pauses/step,
|
||
warm-start rung-A e400 (adapter lr 3e-4; fresh LoRA lr 1e-3, B
|
||
zero-init so step-0 matches item 22 exactly), 200 steps, seed 0.
|
||
Eval n=256: 0:0 (sanity, must equal base), 2:12, 2:16; e200.
|
||
Decision rule (same bands as item 23, vs d=1's 31.6): >=+5
|
||
(>36.6) = band expressivity was binding -> escalate (rank 64, all
|
||
projections, or full-band rung ladder); within +-5 = the medium
|
||
verdict stands even with a trainable band -> internalization line
|
||
CLOSED, program pivots (A2 / E2-N / coarse-aux / divergence
|
||
batch). Prediction, honest: mild gain but under threshold (~33-36)
|
||
— the microscopy says the board's failure mode is WHAT it carries,
|
||
not how flexibly it transforms it; but this is the experiment the
|
||
'wider state' hypothesis deserves before the line closes. Job:
|
||
scripts/jobs/zzz_p_rungb_blr.sh.
|
||
|
||
--- Outcome, item 24 (closed 2026-07-17 ~02:10, STOPPED by Nils
|
||
mid-eval: "stop that idea. does not seem to work either"). Training
|
||
completed; evidence at stop: val easy 0.346 / hard 0.405 vs
|
||
adapter-only 0.356/0.361 — the trainable band did not improve fit
|
||
(hard slightly worse), consistent with the registered prediction that
|
||
capacity/expressivity is not what binds. k=0 sanity row with trained
|
||
LoRA loaded reproduced base EXACTLY (10.9/93.1/0.8) — the loop-only
|
||
LoopLoRA guarantee holds in practice; the mechanism is validated even
|
||
though the hypothesis died (useful for any future band-side work).
|
||
Accuracy cells not measured (eval killed at Nils's call to save ~50min
|
||
GPU); adapter+lora e200 checkpoints in the bucket if anyone ever wants
|
||
the number. Verdict: with items 22+23 this closes the E2-L
|
||
internalization line entirely — adapter capacity, training time,
|
||
pause bandwidth, and band expressivity have all now failed to move
|
||
the d=1 break. The carried state's native cargo (plans, magnitudes,
|
||
completion-state — see the probe series) is the program's remaining
|
||
asset; next candidates: divergence batch replay, coarse-target
|
||
auxiliary supervision, A2, E2-N.
|
||
|
||
25. **E2-L d=1 with latent process supervision through the lens
|
||
(pre-registered 2026-07-17 ~02:45, before running; Nils's idea:
|
||
"for training, i wonder if we could calculate, using jspace lens,
|
||
how each iteration should think").** Items 22-24 all trained the
|
||
latent chain blind — output CE only — and all failed; this changes
|
||
the INFORMATION reaching the chain, not its capacity. New loss:
|
||
the lens readout softmax(W_U·finalnorm(J̄_L30·h)) is differentiable
|
||
in h, so at the 10 replacement pauses we apply lens-CE against the
|
||
DELETED step's tokens, aligned 1:1 (pause j <-> step token j,
|
||
truncated at 10) — the board is trained to write the deleted step
|
||
in lens-readable code at the time it would have been written.
|
||
Mixed loss CE_out + λ·CE_lens. Two arms, single submit: λ=0.3 and
|
||
λ=1.0. Otherwise identical to item-22 d=1 (front-first deletion,
|
||
warm-start rung-A e400, adapter-only 3e-4, 200 steps, seed 0; no
|
||
band-LoRA — one knob). Smoke: step-0 lce=10.3 (~uniform: pauses
|
||
currently encode nothing about the step; large fresh gradient).
|
||
Eval n=256: 0:0, 2:12, 2:16 per arm. Decision vs d=1's 31.6, same
|
||
bands: >36.6 = latent supervision was the missing ingredient ->
|
||
ladder REOPENS with lens-taught rungs (and the 2D per-iteration
|
||
variant becomes item 26); within +-5 = even telling the board
|
||
exactly what to write doesn't make the carry compute it -> the
|
||
strongest closure evidence yet. Caveats pre-stated: J̄ is
|
||
prompt-averaged (global directions); the loss forces a
|
||
verbalizable code (microscopy suggests that IS the board's working
|
||
code, but a native non-verbal code would be fought); the 1:1
|
||
temporal alignment is one choice among several (bag-of-tokens,
|
||
result-digits-only are untested alternatives if this null's).
|
||
Job: scripts/jobs/zzz_q_rungb_lt.sh.
|
||
IN-FLIGHT NOTE (~03:20): λ=0.3 arm's lens-CE collapsed 10.3 -> 2.3
|
||
by step 40 — world 1 (cannot encode) is eliminated before the eval;
|
||
the open question is world 2 (writes but doesn't compute) vs
|
||
world 3 (reopens the ladder).
|
||
|
||
--- Outcome, item 25 (closed 2026-07-17 ~01:25, TRUNCATED by Nils:
|
||
"cancel that 2:16 and then try the new approach"). λ=0.3 arm: lens-CE
|
||
converged to ~1.9 (the pause-chain reliably writes the deleted step in
|
||
lens code, top-few over 262k vocab) while eval 2:12 = 31.2
|
||
(drop .18 / easy .586 / hard .354) vs the 31.6 reference — FLAT, not
|
||
even a redistribution across buckets. k=0 sanity clean. World 2
|
||
confirmed on the measured cell: WRITING IS NOT COMPUTING — the
|
||
carried state can be taught to verbalizably display the missing step
|
||
at the right timesteps and the displayed content is not consumed
|
||
downstream. 2:16 and the λ=1.0 arm cancelled mid-run (λ=0.3's flat
|
||
cell made λ=1.0 uninformative; ~2.5h GPU saved). Program value: the
|
||
cleanest representation/computation dissociation in the series; the
|
||
read-side (what the suffix computation consumes from the board) is
|
||
now the prime suspect, and the pause-tape architecture allows
|
||
display-without-routing. Item 27 (zero-pause internal looping, no
|
||
tape to display on) runs next by Nils's direction; item 26 requeued
|
||
after it (job renamed zzz_t).
|
||
|
||
26. **Result-staging supervision during generation (pre-registered
|
||
2026-07-17 ~03:25, before running; Nils: "can we try to train the
|
||
loop using jspace during generation?").** The microscopy's
|
||
signature of the healthy board: each line's RESULT is on the
|
||
carried state before it is emitted ("1/twelve" at the '=',
|
||
P=0.997). This trains that signature directly. For every visible
|
||
scratchpad line, at the pre-'=' positions (the result is NOT yet
|
||
in the causal context there — nothing can be copied; low loss
|
||
requires computation) lens-CE the L30 carried state against the
|
||
line's result tokens; Answer-line analog via the ':' separator.
|
||
Spans computed in token space (gen_staging_targets; 1102 spans on
|
||
the d=1-transformed 427). Two arms, one submit, both otherwise
|
||
item-22 d=1 (warm-start rung-A e400, 200 steps, adapter-only):
|
||
(a) lg=0.3 alone — staging supervision without pause supervision;
|
||
(b) lt=0.3 + lg=0.3 — the full lens curriculum (deleted step
|
||
taught at the pauses, results staged during emission). Eval n=256:
|
||
0:0, 2:12, 2:16 per arm; decision vs 31.6, same bands as items
|
||
23-25. Distinct from item 25's question: 25 teaches WHAT the
|
||
silent chain should hold; 26 teaches the compute-before-emit
|
||
discipline on the visible chain (the behavior that separated
|
||
carry from FF at the 430 divergence). Caveat: on-policy ("during
|
||
generation" in the free-running sense) is NOT this — that's a
|
||
rollout-based follow-up (latent DAgger) if either lens arm moves.
|
||
Job: scripts/jobs/zzz_r_rungb_lg.sh.
|
||
|
||
--- Outcome, item 26 (scored 2026-07-17 ~10:50; ran last, requeued as
|
||
zzz_t after items 27/28 per Nils's mid-night reprioritization). k=0
|
||
sanity clean both arms. Arm a (staging-only, lg03): 24.2 / 23.4 —
|
||
the series' first ACTIVELY NEGATIVE result, ~7 points below the 31.6
|
||
reference, outside the band on the downside (easy 44.8-48.3, hard
|
||
~26-28, drop 12-15). Arm b (combined lt03+lg03): 15.6 / 14.1 —
|
||
compounding destruction (easy 27.6, hard 18-21, drop 5-6), worse than
|
||
either constraint alone. Reading: the microscopy showed the healthy
|
||
board computes results 1-2 positions pre-emission (just-in-time); the
|
||
staging loss demanded the result across the ENTIRE pre-'=' span,
|
||
including positions where operands haven't been read — forcing
|
||
thought earlier than the computation can exist corrupts the schedule
|
||
rather than accelerating it. Lesson for any future lens supervision:
|
||
match the loss's temporal demand to the board's measured schedule
|
||
(supervise at the '=' +- 1 position only), don't blanket-force.
|
||
Overnight program complete: items 25-28 all scored; write-side
|
||
exhausted; the standing decision (clamp test vs pivot to hybrid/A2)
|
||
is unchanged by this result.
|
||
|
||
27. **Internal band looping, zero pause tokens (pre-registered
|
||
2026-07-17 ~01:10, before running; Nils's clarified structure,
|
||
option A confirmed via question: one silent burst before output).**
|
||
Architecture: NO pause tokens anywhere — sequence is prompt +
|
||
visible output only. After the k=2 prompt settle, the last prompt
|
||
position runs M=10 IN-PLACE band iterations (s^i seeds from the
|
||
position's own previous band output — carry_common inplace
|
||
updates), then generation proceeds with the normal single-pass
|
||
carry under the visible tokens. Vertical thought burst instead of
|
||
a horizontal pause tape: iterations leave no KV entries — only
|
||
the final state survives, a true internal loop (the C'-flavored
|
||
architecture the ladder never tested; every prior rung bought
|
||
compute with positions). Supervision: lens-CE iteration i <->
|
||
deleted-step token i (λ=0.3, the stable value), same d=1 deletion,
|
||
warm-start rung-A e400, 200 steps, seed 0. Eval n=256: 0:0
|
||
(sanity), 2:0 with --inner-iters 10 (matched), 2:0 with
|
||
--inner-iters 0 (ablation: does the burst matter at inference?).
|
||
References: positional d=1 31.6 (same ±5 / >36.6 bands) and the
|
||
ii0 ablation cell (isolates the burst's causal contribution).
|
||
Honest priors: item 25's 2:12 just landed at 31.2 (flat), so the
|
||
write-vs-compute dissociation likely carries over; the
|
||
architectural deltas that could matter here: no attention-tape
|
||
(forces state-borne computation rather than KV re-reading) and
|
||
anchor-at-prompt (iterations see the full settled question).
|
||
Job: scripts/jobs/zzz_s_rungb_ii.sh.
|
||
IN-FLIGHT NOTE (~02:20): lce 9.4 -> 2.9 by step 60 — the tape-free
|
||
burst also encodes the step trajectory; encoding is never the
|
||
obstacle. Accuracy pending.
|
||
|
||
--- Outcome, item 27 (scored 2026-07-17 ~04:35). k=0 sanity clean.
|
||
Matched cell 2:0 (burst on) = 34.0 (drop .19 / easy .621 / hard .394)
|
||
— nominally the BEST d=1 number of the series (vs 31.6 positional,
|
||
31.2 lens-pauses) with ZERO pause tokens; ablation (same adapter,
|
||
burst off) = 32.4. But both pre-registered comparisons are
|
||
noise-compatible when paired: burst-on vs burst-off 10-6 discordants,
|
||
p=0.45; vs positional d=1 44-38, p=0.58. Verdict: within the +-5
|
||
band — the ceiling holds on the internal-iteration axis too; the
|
||
burst's causal contribution is directionally positive (all of it in
|
||
the hard bucket, +3.2) but unproven at n=256. Notable anyway: the
|
||
zero-pause FORMAT loses nothing (ablation 32.4 >= positional 31.6 with
|
||
12 fewer latent positions) — the pause tape was contributing ~nothing,
|
||
consistent with item 25's display-without-routing reading. Item 28
|
||
(teacher-state targets, same architecture) runs next and completes
|
||
the 2x2 (verbal/full-state x tape/no-tape).
|
||
|
||
28. **Teacher-state distillation into the burst (pre-registered
|
||
2026-07-17 ~02:30, before running; Nils's variant: "meaningful CoT
|
||
chunks yield internal state that we then teacher-force into the
|
||
loop-only model").** Items 25/27 supervise the VERBAL SHADOW of
|
||
the deleted step (token targets through the lens); this forces the
|
||
FULL state. Teacher = the frozen warm-start (rung-A e400) adapter
|
||
running the complete cot (deleted step visible), zero pauses, same
|
||
carry architecture; capture its band-exit state at the deleted
|
||
step's LAST token — "the state of having finished thinking the
|
||
step" — one no-grad pass over the 413 parseable items at startup
|
||
(targets fixed, no moving teacher). Student: identical zero-pause
|
||
M=10 burst as item 27, but the loss is cosine distance between the
|
||
burst's FINAL iterate s^10 and the teacher state, plus output CE.
|
||
Weight AMENDED pre-run λ=1.0 -> 5.0 after the smoke measured the
|
||
starting cosine distance at only 0.113 (nearby band-exit states
|
||
share most structure); at 1.0 the term would be ~10x weaker than
|
||
the CE and a null would be underpowered — 5x0.113 puts the two
|
||
terms at comparable initial scale. No lens loss (one knob vs item 27: full-state
|
||
targets instead of verbal-shadow targets; trajectory supervision
|
||
dropped — only the endpoint is forced). Known approximation,
|
||
stated in advance: teacher state is captured ~10 positions later
|
||
in the sequence than the student anchor (RoPE position coloring);
|
||
cosine + the carry machinery's routine state transplantation
|
||
across nearby positions make this tolerable, but a null could
|
||
partly hide here. Eval n=256: 0:0, 2:0 ii10 (matched), 2:0 ii0
|
||
(ablation). Decision vs 31.6, same bands; the informative
|
||
three-way is 25 (verbal, tape) vs 27 (verbal, no tape) vs 28
|
||
(full state, no tape) — if 28 moves where 25/27 don't, the
|
||
computation-carrying content lives OUTSIDE the verbalizable
|
||
subspace; if none move, the read-side clamp test (item 29
|
||
candidate) decides. Job: scripts/jobs/zzz_sa_rungb_ts.sh.
|
||
|
||
--- Outcome, item 28 (scored 2026-07-17 ~06:20). k=0 sanity clean.
|
||
Distillation SUCCEEDED geometrically: cos-dist 0.113 -> 0.044 (burst
|
||
endpoint lands ~60% closer to the teacher state), with the lowest
|
||
output-CE tax of any supervised arm (vals 0.384/0.376). Function did
|
||
NOT follow: matched 2:0 = 30.1, ablation (burst off) = 29.7 — burst
|
||
contribution nil; hard 34.6 indistinguishable from the pack; and the
|
||
state-forcing DAMAGED easy (48.3 vs 62.1 in every other d=1 arm —
|
||
the forced endpoint displaces something easy handling needed).
|
||
Verdict: null, below reference. The 2x2 (items 25/27/28 + 22 as the
|
||
unsupervised corner) is complete and uniform: no state-side
|
||
supervision — verbal shadow or full teacher state, tape or burst —
|
||
makes the loop compute the deleted step. Combined with items 22-24
|
||
(capacity/time/positions/expressivity all null), the write-side is
|
||
exhausted end to end. The one untested link is READ-side: whether the
|
||
suffix computation consumes the workspace at all when the right state
|
||
is present — the clamp test (write the teacher state into the anchor
|
||
at inference via the lens/embedding-basis machinery, no training, and
|
||
measure) is item 29's natural content and would assign blame causally.
|
||
Nils's morning decision: clamp test vs pivot to the hybrid/A2 line
|
||
(where the 57.4 and the drop-bucket reach already live).
|
||
|
||
29. **Trajectory teacher-forcing of the burst (pre-registered
|
||
2026-07-17 ~11:50, before running; Nils's design: "measure the CoT
|
||
run's L30 at 10 evenly spaced intervals and teacher force them
|
||
into the loop — we know both the previous that goes into the merge
|
||
layer and the next L30 state").** The key upgrade over item 28:
|
||
the teacher trajectory decomposes the burst's learning into TEN
|
||
INDEPENDENT SUPERVISED TRANSITIONS — iteration i receives teacher
|
||
waypoint T[i-1] as its merge input and its band output is pulled
|
||
onto T[i] (cosine, λ=5.0) — dense gradient at every iteration, no
|
||
long BPTT chain, no compounding; and T[0] == the student's own
|
||
settled anchor state exactly (shared prompt+settle), so the chain
|
||
starts aligned by construction. Exposure gap handled by a second
|
||
arm adding the free-running loss (burst's own s_i onto T[i],
|
||
λ=5.0). The free-running burst ALWAYS runs last and seeds the
|
||
answer scan (training matches inference); TF transitions are
|
||
scaffolding erased from context afterward. Teacher = frozen
|
||
warm-start adapter on the full cot, 10 evenly spaced L30 states.
|
||
Two jobs, per Nils (both chosen): JOB 1 (zzz_u) d=1, waypoints
|
||
across the deleted step, arms tf-only and tf+fr; reference 31.6,
|
||
same bands; eval 0:0 / 2:0 ii10 / 2:0 ii0 per arm. JOB 2 (zzz_v)
|
||
THE HEADLINE: d=all (answer-only output, rung C), waypoints across
|
||
the FULL CoT — the burst as a 10-waypoint compressed rehearsal of
|
||
the entire reasoning; references: cold answer-only 9.4, curriculum
|
||
plateau 19.1, base 10.9; same arms and cells. Smokes: ltf/lfr
|
||
baseline 0.116 both modes; d=99 deletes 1118 steps across 427.
|
||
Priors, stated: the transition-decomposition is the first
|
||
mechanism that trains the loop's DYNAMICS rather than its
|
||
endpoints — if the loop can compute at all, this is its best
|
||
chance; if job 2 beats 19.1 meaningfully, compressed-trajectory
|
||
rehearsal becomes the internalization method. All prior caveats
|
||
(position coloring, teacher=warm-start quality) carry over.
|
||
Jobs: scripts/jobs/zzz_u_traj_d1.sh, scripts/jobs/zzz_v_traj_full.sh.
|