integrate LCB dissociation (untrained merge wins far-transfer, p=0.001 vs trained loop), capability panel (no MC damage), L23 exit completes sweep

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

LESSONS.md

@@ -76,3 +76,13 @@ reproduction: [`RESULTS.md`](RESULTS.md). Everything on `google/gemma-4-E2B-it`
 3. **Verify with the lens, gate with the labels**: the J-lens picks the band,
    measures whether loops compute, and diagnoses failures; STaR difficulty
    labels supervise both the curriculum and (next) the adaptive-depth gate.
+
+## 12. Pin model revisions on fresh nodes
+Upstream updated google/gemma-4-12B-it mid-project: the new chat template
+adds a `<|channel>thought` scaffold, and greedy generation closes the empty
+thought channel and stops — every generation decodes to "". Symptom:
+0/500 pass rates with rc=0 (looks like a harness bug, is a silent model
+swap). E2B was unaffected. Fix: `hf download --revision <hash>` + repoint
+`refs/main` in the cache; node_setup.sh now pins both models (12B
+0e2b1058…, E2B 9dbdf8a8…). Rule: any cross-node result assumes identical
+model revisions — pin them, don't trust "main".

PAPER.md

@@ -31,10 +31,15 @@ A complete attribution program then deflates the mechanism's mystique: the
 loop's content is *amortizable* (plan-distillation into the same adapter,
 recurrence-free, matches it; nothing stacks; looping distilled weights
 degrades them), width rivals depth (trained pause registers reach 36.4%),
-and compute-matched token baselines win overall accuracy outright. What
+and verifier-assisted (oracle) sampling wins overall accuracy at matched
+compute — though the *deployable* selector loses that edge entirely. What
 survives is precise: the implant owns exactly the plan-dependent slice at
-zero visible tokens and zero decode cost, transfers with the substrate
+zero visible tokens and zero decode cost, and its placement is dictated by
-rather than the task, and its placement is dictated by the lens. At 12B a
+the lens. Transfer dissociates by distance: near-distribution the trained
+and untrained implants tie (HumanEval); far from it (LiveCodeBench) the
+*untrained* merge significantly helps while the trained content
+significantly hurts — the learned content is task-local, the recurrence
+substrate is general. At 12B a
 constant merge coefficient destroys the substrate; a state-dependent
 coefficient (3.8K parameters) restores MBPP but not Blocksworld or GSM8K —
 the anchor coefficient is the stability dial that unifies this work with
@@ -88,11 +93,14 @@ not laws.)*
    (parallel, prefill-shaped), zero decode overhead, bit-exact KV-cache
    write-in, k=0 recovers the base model exactly. Its competition at
    comparable compute (accounting in Appendix A — FLOPs, wall-clock, and
-   token budget do not rank the arms the same way): best-of-3 sampling wins
+   token budget do not rank the arms the same way): *oracle* best-of-3
-   overall accuracy outright (57.2%); a
+   (verifier-assisted) wins overall accuracy (57.8%, paired p=0.020 vs
-   50-token visible plan ties the hard bucket. The value proposition is
+   loop), but the *deployable* logprob-selected variant drops to 55.0%
-   *only*: no visible tokens, no decode latency, and the hard-slice
+   overall / 27.3% hard — indistinguishable from the latent arms overall
-   specialization (distill's 46% > budget-CoT's 40% > best-of-3's 33%).
+   and directionally behind on hard. A 50-token visible plan ties the hard
+   bucket. The value proposition without a verifier: no visible tokens, no
+   decode latency, and the hard-slice specialization (distill's 46% >
+   budget-CoT's 40% > deployable sampling's 27%).
 
 6. **Scale transfers only with a state-dependent stability dial.** At 12B the
    2B-tuned constant α=0.3 collapses overall accuracy (72.6%→43.0%); the
@@ -170,8 +178,9 @@ hard 43.6%, overall 53.6%. L13: hard 17.9%, overall 34.4%. L9–L12: overall
 verified bit-identical) — the 12B model has no shared-KV layers, making it
 the unconfounded replication. Exit sweep at fixed entrance (L27/30/32/34):
 hard 39.3–46.4%, within seed spread. The lens boundary is necessary; the
-exit is a free parameter. (The L23-exit arm died in training; a rerun is in
+exit is a free parameter — the completed five-point exit sweep
-progress — the four completed exits bracket it. [L23 PENDING])
+(L23/27/30/32/34) spans hard 39.3–46.4% with L23 at the top (46.4% at k=2),
+all within seed spread.
 
 ### 3.2 The attribution ladder
 
@@ -228,20 +237,29 @@ distinctive value is that it finds this content *without* plan supervision
 ### 3.4 Compute-matched honesty
 
 At approximately matched FLOPs (Appendix A gives the accounting, separated
-into FLOPs, wall-clock, and token budget), token-space baselines are strong:
+into FLOPs, wall-clock, and token budget), the token-space comparison
-best-of-3 sampling wins overall accuracy against every latent arm (57.2%,
+splits into two very different claims:
-CI [52.8, 61.5], vs loop 53.6 [49.2, 57.9] — point estimate higher, CIs
+
-overlap) by preserving easy items perfectly while sampling rescues some hard
+| best-of-3 variant | overall | hard | vs loop (paired) |
-ones. A 50-token visible plan ties the loop's hard bucket. Both baselines
+|---|---|---|---|
-are being rerun with per-item logs to enable paired tests against the latent
+| **oracle** (any-of-3 passes; needs a perfect verifier) | 57.8% | 34.5% | beats loop overall, p=0.020 |
-arms; until those land, the overall-accuracy comparison rests on overlapping
+| **deployable** (highest mean logprob of 3) | 55.0% | 27.3% | n.s. overall (p=0.47); loop ahead on hard 16–7 (p=0.09) |
-CIs and is stated as point-estimate-level. [BASELINES-PI PENDING]
+
-The latent implant's surviving advantages are qualitative: zero visible
+The earlier draft's "sampling wins overall" was the **oracle** number — an
-tokens (silent), zero decode overhead (prefill-parallel; sampling and CoT
+upper bound requiring an external verifier that MBPP's own tests provide
-pay serially at bandwidth-bound decode), and the hard-slice crown under
+but a deployment does not. With the deployable selector (identical seeded
-distillation (46% vs 40% budget-CoT vs 33% best-of-3). For deployment this
+samples, so the comparison is exact), best-of-3 is statistically
-means: the implant is a *latency/token-budget* technology with a side
+indistinguishable from the latent arms overall, *behind* them
-specialization in plan-dependent items — not an accuracy technology.
+directionally on the hard bucket, and pays ≈3× visible tokens and serial
+decode for it. Budget-CoT-50 remains the strongest honest token baseline
+(53.8% overall, hard 40.0%; per-item rerun 53.8/38.2) — and the paired
+tests confirm it is a *tie* with the latent arms on both axes (p≥0.69 vs
+loop and distill), at the price of 50 visible tokens and their serial
+decode latency. The implant's advantages at matched compute
+without a verifier: zero visible tokens, zero decode overhead, and the
+hard-slice crown under distillation (46%). Where a task *does* come with a
+cheap verifier, oracle-style sampling is the better overall-accuracy
+spend — both halves belong in the deployment picture.
 
 ### 3.5 Width vs depth, and the task boundary
 
@@ -284,7 +302,28 @@ significant** (paired McNemar at k=2, 9 vs 7 discordant, p=0.80). What
 transfers significantly is the *merge perturbation itself*, not the
 MBPP-trained content — the cleanest evidence that off-distribution value is
 substrate-shaped rather than task-memorized. (The transferred pause adapter
-reaches 66.5%, hard 38.9%, consistent with the same reading.) **Rust/MultiPL-E** (Python-trained, different
+reaches 66.5%, hard 38.9%, consistent with the same reading.)
+
+**LiveCodeBench sharpens this into a dissociation** (150 newest stdin
+problems, Nov 2024–Apr 2025, execution-verified; no LCB training anywhere
+in the pipeline; base 18.7%):
+
+| arm (MBPP-trained where trained) | overall | hard (n=25) | vs base, paired |
+|---|---|---|---|
+| **untrained merge, k=4** | **24.0%** | **36.0%** | **+**, p=0.039 |
+| trained loop, k=4 | 15.3% | 8.0% | −, p=0.23 |
+| distill FF, k=1 | 12.7% | 16.0% | **−**, p=0.049 |
+
+Far from distribution, the *trained content is a liability* (distill
+significantly hurts; untrained-vs-trained-loop is 14–1 discordant,
+p=0.001) while the *untrained anchored recurrence significantly helps* —
+the training-free regime of Lys et al. is the right choice off-distribution,
+and the amortized-content reading of §3.3 predicts exactly this: what the
+adapter learned is MBPP-shaped plan content, valuable where plans look like
+MBPP plans and harmful where they don't. Transfer ordering by distance:
+HumanEval (near) — trained ≈ untrained; Rust (mid) — trained helps the hard
+bucket; LCB (far) — untrained wins outright. Caveats: single seed per arm,
+hard n=25, one benchmark at the far end. **Rust/MultiPL-E** (Python-trained, different
 language, compile-run-verified): hard 8.0%→24.0% (p=0.125 at n=25 —
 directionally consistent, underpowered). **Blocksworld** MBPP-transfer:
 hard 0→14.3% (task-trained: 43%). Content transfers where the substrate's
@@ -303,9 +342,22 @@ removing the easy-item perturbation tax. k=0 is the exact base model by
 construction — the implant is removable at token granularity.
 
 **General-capability panel** (ARC-Challenge, WinoGrande, HellaSwag, MMLU;
-length-normalized MC scoring with the loop applied to the context span) is
+800 items each, length-normalized MC likelihood via the chat template, loop
-running on the Spark; results will quantify what k>0 does to off-task
+applied to the context span). The safety answer is clean — **k>0 does not
-abilities. [PENDING — fill on completion.]
+damage general abilities**:
+
+| arm | ARC-C | WinoGrande | HellaSwag | MMLU |
+|---|---|---|---|---|
+| base (k=0) | 36.0 | 55.9 | 52.3 | 30.1 |
+| loop k=2 (MBPP adapter) | 36.1 | 55.3 | 49.6 | 31.3 |
+| distill FF (MBPP) | 41.8 | 56.6 | 57.0 | 31.8 |
+
+The loop arm is flat within noise (largest move −2.6 on HellaSwag,
+unpaired n=800). The distill adapter *nominally improves* every benchmark
+(+5.8 ARC, +4.8 HellaSwag) — consistent with §3.7's finding that these
+implants carry a generically useful perturbation component, though
+MC-likelihood scoring and generation quality are different regimes (see
+the LCB result below before reading this as free capability).
 
 ### 3.9 Negative results with content
 
@@ -361,9 +413,15 @@ are small (n=55/38/25); within-ladder orderings are not individually
 significant, and only the pooled hard effect and the HumanEval overall gain
 survive multiple-comparison scrutiny. Bucket membership derives from greedy
 labeling runs (consensus-k0 robustness check moves numbers <2 points, but
-both checks share the base model; an independent difficulty proxy is an
+both checks share the base model; an independent 12B-relabeling proxy is
-open external check). A third architecture family was not run;
+running). A third architecture family was not run; rung-2 was not run at
-LiveCodeBench (contamination-safe) was not run; rung-2 was not run at 12B. The easy-item perturbation tax persists wherever the
+12B. LiveCodeBench: single seed per arm, hard n=25, stdin-judged problems
+only, and its newest shard (Apr 2025) is *newer than MBPP by years* but
+not provably past the base model's undisclosed training cutoff — we claim
+recency, not proven non-contamination. The capability panel is
+MC-likelihood, not generation; its "no damage" answer does not extend to
+generation quality off-distribution (LCB shows trained arms *do* hurt
+there). The easy-item perturbation tax persists wherever the
 gate's precision fails. MBPP/GSM8K likely overlap pretraining data; both
 arms share contamination, and memorized items land in the easy bucket, but
 bucket composition is contamination-sensitive. The capability panel

results-loop/STATS.md

@@ -31,6 +31,25 @@
 - Rust loop k=4 vs k=0, overall: A-only 11, B-only 5, n=154, p=0.2101 (n.s.)
 - Rust loop k=4 vs k=0, hard: A-only 0, B-only 4, n=25, p=0.125 (n.s.)
 
+## Token baselines (paired, per-item)
+
+- **best-of-3 ORACLE (any-pass)**: overall 0.578 [0.534, 0.621] (n=500); hard 0.345 [0.234, 0.477] (n=55)
+- **best-of-3 oracle (selector run)**: overall 0.578 [0.534, 0.621] (n=500); hard 0.345 [0.234, 0.477] (n=55)
+- **best-of-3 DEPLOYABLE (logprob-selected)**: overall 0.550 [0.506, 0.593] (n=500); hard 0.273 [0.173, 0.402] (n=55)
+- bo3-oracle vs loop k=4, overall: arm-only 27, bo3-only 48, p=0.0203 (**significant**)
+- bo3-oracle vs loop k=4, hard: arm-only 14, bo3-only 9, p=0.4049 (n.s.)
+- bo3-oracle vs distill k=1, overall: arm-only 27, bo3-only 44, p=0.05681 (n.s.)
+- bo3-oracle vs distill k=1, hard: arm-only 14, bo3-only 10, p=0.5413 (n.s.)
+- bo3-deployable vs loop k=4, overall: arm-only 31, bo3-only 38, p=0.4704 (n.s.)
+- bo3-deployable vs loop k=4, hard: arm-only 16, bo3-only 7, p=0.09314 (n.s.)
+- bo3-deployable vs distill k=1, overall: arm-only 31, bo3-only 34, p=0.8043 (n.s.)
+- bo3-deployable vs distill k=1, hard: arm-only 15, bo3-only 7, p=0.1338 (n.s.)
+- **budget-CoT-50 (per-item rerun)**: overall 0.538 [0.494, 0.581] (n=500); hard 0.382 [0.265, 0.514] (n=55)
+- budget-CoT vs loop k=4, overall: arm-only 36, cot-only 37, p=1 (n.s.)
+- budget-CoT vs loop k=4, hard: arm-only 14, cot-only 11, p=0.69 (n.s.)
+- budget-CoT vs distill k=1, overall: arm-only 29, cot-only 26, p=0.7877 (n.s.)
+- budget-CoT vs distill k=1, hard: arm-only 8, cot-only 6, p=0.7905 (n.s.)
+
 ## Pooled hard bucket (MBPP + HumanEval + Rust)
 
 Paired within-item k>0 vs k=0, counts pooled across benchmarks (loop arm; distill pooled where available).

results-loop/eval_panel_base_k0.json

@@ -0,0 +1,8 @@
+{
+ "tag": "base_k0",
+ "k": 0,
+ "arc": 0.36,
+ "winogrande": 0.55875,
+ "hellaswag": 0.5225,
+ "mmlu": 0.30125
+}

results-loop/eval_panel_distill_ff.json

@@ -0,0 +1,8 @@
+{
+ "tag": "distill_ff",
+ "k": 1,
+ "arc": 0.4175,
+ "winogrande": 0.56625,
+ "hellaswag": 0.57,
+ "mmlu": 0.3175
+}

results-loop/eval_panel_loop_k2.json

@@ -0,0 +1,8 @@
+{
+ "tag": "loop_k2",
+ "k": 2,
+ "arc": 0.36125,
+ "winogrande": 0.5525,
+ "hellaswag": 0.49625,
+ "mmlu": 0.3125
+}

results-loop/lcb_labels.json

@@ -0,0 +1,152 @@
+{
+ "abc382_a": true,
+ "abc382_b": true,
+ "abc382_c": false,
+ "abc382_d": true,
+ "abc382_f": false,
+ "abc382_g": false,
+ "abc383_a": true,
+ "abc383_b": true,
+ "abc383_c": false,
+ "abc383_d": true,
+ "abc383_e": false,
+ "abc384_a": true,
+ "abc384_b": true,
+ "abc384_c": true,
+ "abc384_d": false,
+ "abc384_e": false,
+ "abc384_f": false,
+ "abc384_g": false,
+ "abc385_a": true,
+ "abc385_b": true,
+ "abc385_c": false,
+ "abc385_d": false,
+ "abc385_e": false,
+ "abc385_f": false,
+ "abc386_a": false,
+ "abc386_b": true,
+ "abc386_c": false,
+ "abc386_d": false,
+ "abc386_e": false,
+ "abc386_f": false,
+ "abc387_a": true,
+ "abc387_b": true,
+ "abc387_c": false,
+ "abc387_f": false,
+ "abc388_a": true,
+ "abc388_b": true,
+ "abc388_c": false,
+ "abc388_d": false,
+ "abc388_e": false,
+ "abc388_f": false,
+ "abc388_g": false,
+ "abc389_a": true,
+ "abc389_b": true,
+ "abc389_d": false,
+ "abc389_e": false,
+ "abc389_f": false,
+ "abc389_g": false,
+ "abc390_a": true,
+ "abc390_b": true,
+ "abc390_c": false,
+ "abc390_d": false,
+ "abc390_e": false,
+ "abc390_f": false,
+ "abc390_g": false,
+ "abc391_a": true,
+ "abc391_b": true,
+ "abc391_d": false,
+ "abc391_e": false,
+ "abc391_f": false,
+ "abc391_g": false,
+ "abc392_a": true,
+ "abc392_b": true,
+ "abc392_c": false,
+ "abc392_d": false,
+ "abc392_f": true,
+ "abc392_g": false,
+ "abc393_a": true,
+ "abc393_b": true,
+ "abc393_d": false,
+ "abc393_e": true,
+ "abc393_f": false,
+ "abc394_a": true,
+ "abc394_b": true,
+ "abc394_c": true,
+ "abc394_d": true,
+ "abc394_e": false,
+ "abc394_f": false,
+ "abc394_g": false,
+ "abc395_a": true,
+ "abc395_b": true,
+ "abc395_c": true,
+ "abc395_e": true,
+ "abc395_f": false,
+ "abc396_a": true,
+ "abc396_b": true,
+ "abc396_c": false,
+ "abc396_d": true,
+ "abc396_e": false,
+ "abc396_f": false,
+ "abc396_g": false,
+ "abc397_a": true,
+ "abc397_b": false,
+ "abc397_c": true,
+ "abc397_d": false,
+ "abc397_e": false,
+ "abc397_f": false,
+ "abc397_g": false,
+ "abc398_a": true,
+ "abc398_b": false,
+ "abc398_c": true,
+ "abc398_d": false,
+ "abc398_f": true,
+ "abc398_g": false,
+ "abc399_a": true,
+ "abc399_b": true,
+ "abc399_c": true,
+ "abc399_d": false,
+ "abc399_e": false,
+ "abc399_f": false,
+ "abc400_a": true,
+ "abc400_b": true,
+ "abc400_c": false,
+ "abc400_d": false,
+ "abc400_e": false,
+ "abc400_g": false,
+ "arc188_a": false,
+ "arc188_b": false,
+ "arc188_c": false,
+ "arc188_d": false,
+ "arc189_a": false,
+ "arc189_b": false,
+ "arc189_c": false,
+ "arc189_d": false,
+ "arc190_a": false,
+ "arc190_c": false,
+ "arc190_d": false,
+ "arc191_a": false,
+ "arc191_c": false,
+ "arc191_d": true,
+ "arc192_a": false,
+ "arc192_b": false,
+ "arc192_d": false,
+ "arc192_e": false,
+ "arc193_a": false,
+ "arc193_b": false,
+ "arc193_d": false,
+ "arc194_a": false,
+ "arc194_b": false,
+ "arc194_c": false,
+ "arc194_d": false,
+ "arc194_e": false,
+ "arc195_a": false,
+ "arc195_b": false,
+ "arc195_c": false,
+ "arc195_d": false,
+ "arc195_e": false,
+ "arc196_a": false,
+ "arc196_b": false,
+ "arc196_c": false,
+ "arc196_d": false
+}

results-loop/stats_final.json

@@ -322,6 +322,132 @@
    "a_only": 0,
    "b_only": 4,
    "p": 0.125
+  },
+  "bo3-oracle vs loop k=4, overall": {
+   "n": 500,
+   "a_only": 27,
+   "b_only": 48,
+   "p": 0.020298406990107445
+  },
+  "bo3-oracle vs loop k=4, hard": {
+   "n": 55,
+   "a_only": 14,
+   "b_only": 9,
+   "p": 0.4048728942871094
+  },
+  "bo3-oracle vs distill k=1, overall": {
+   "n": 500,
+   "a_only": 27,
+   "b_only": 44,
+   "p": 0.056814677932839015
+  },
+  "bo3-oracle vs distill k=1, hard": {
+   "n": 55,
+   "a_only": 14,
+   "b_only": 10,
+   "p": 0.5412561893463135
+  },
+  "bo3-deployable vs loop k=4, overall": {
+   "n": 500,
+   "a_only": 31,
+   "b_only": 38,
+   "p": 0.4703685318581444
+  },
+  "bo3-deployable vs loop k=4, hard": {
+   "n": 55,
+   "a_only": 16,
+   "b_only": 7,
+   "p": 0.0931396484375
+  },
+  "bo3-deployable vs distill k=1, overall": {
+   "n": 500,
+   "a_only": 31,
+   "b_only": 34,
+   "p": 0.8043170001933986
+  },
+  "bo3-deployable vs distill k=1, hard": {
+   "n": 55,
+   "a_only": 15,
+   "b_only": 7,
+   "p": 0.13380050659179688
+  },
+  "budget-cot vs loop k=4, overall": {
+   "n": 500,
+   "a_only": 36,
+   "b_only": 37,
+   "p": 1.0
+  },
+  "budget-cot vs loop k=4, hard": {
+   "n": 55,
+   "a_only": 14,
+   "b_only": 11,
+   "p": 0.6900379657745361
+  },
+  "budget-cot vs distill k=1, overall": {
+   "n": 500,
+   "a_only": 29,
+   "b_only": 26,
+   "p": 0.7877061896700435
+  },
+  "budget-cot vs distill k=1, hard": {
+   "n": 55,
+   "a_only": 8,
+   "b_only": 6,
+   "p": 0.79052734375
+  }
+ },
+ "best-of-3 ORACLE (any-pass)": {
+  "overall": {
+   "acc": 0.578,
+   "n": 500,
+   "ci": [
+    0.534,
+    0.621
+   ]
+  },
+  "hard": {
+   "acc": 0.34545454545454546,
+   "n": 55,
+   "ci": [
+    0.234,
+    0.477
+   ]
+  }
+ },
+ "best-of-3 oracle (selector run)": {
+  "overall": {
+   "acc": 0.578,
+   "n": 500,
+   "ci": [
+    0.534,
+    0.621
+   ]
+  },
+  "hard": {
+   "acc": 0.34545454545454546,
+   "n": 55,
+   "ci": [
+    0.234,
+    0.477
+   ]
+  }
+ },
+ "best-of-3 DEPLOYABLE (logprob-selected)": {
+  "overall": {
+   "acc": 0.55,
+   "n": 500,
+   "ci": [
+    0.506,
+    0.593
+   ]
+  },
+  "hard": {
+   "acc": 0.2727272727272727,
+   "n": 55,
+   "ci": [
+    0.173,
+    0.402
+   ]
   }
  },
  "pooled_hard_loop": {

scripts/figures_final.py

@@ -220,8 +220,14 @@ def fig_transfer():
             ("base", *ev(OUT / "eval_rust_py_transfer.json", 0)),
             ("loop k=4", *ev(OUT / "eval_rust_py_transfer.json", 4)),
         ]),
+        ("LiveCodeBench (far)\nuntrained wins", [
+            ("base", *he_hard("eval_lcb_untrained.json", 0)),
+            ("untrained k=4", *he_hard("eval_lcb_untrained.json", 4)),
+            ("loop k=4", *he_hard("eval_lcb_loop.json", 4)),
+            ("distill", *he_hard("eval_lcb_distill.json", 1)),
+        ]),
     ]
-    fig, axes = plt.subplots(1, 2, figsize=(7.6, 3.8))
+    fig, axes = plt.subplots(1, 3, figsize=(10.8, 3.8))
     for ax, (title, arms) in zip(axes, groups):
         style(ax)
         x = range(len(arms))

scripts/node_setup.sh

@@ -10,11 +10,16 @@ rclone copy jspace:jspace/results-loop /dev/shm/jspace/results-loop/ --include "
 export HF_HUB_ENABLE_HF_TRANSFER=1
 # downloads can hang mid-transfer on some vast.ai networks (xet stall):
 # retry loop, resume is free
-for m in google/gemma-4-12B-it google/gemma-4-E2B-it; do
+# PIN revisions: upstream updated gemma-4-12B-it's chat template (added a
+# thought channel) mid-project; unpinned downloads silently change behavior.
+for spec in "google/gemma-4-12B-it 0e2b1058541244490925fbacf8972041435691ac" \
+            "google/gemma-4-E2B-it 9dbdf8a839e4e9e0eb56ed80cc8886661d3817cf"; do
+  set -- $spec
   for i in 1 2 3 4 5; do
-    timeout 900 hf download "$m" && break
+    timeout 900 hf download "$1" --revision "$2" && break
-    echo "RETRY $i: $m"; sleep 5
+    echo "RETRY $i: $1"; sleep 5
   done
+  echo "$2" > "$HF_HOME/hub/models--${1//\//--}/refs/main"
 done
 # probe gate: one verified generation before anything batch
 export HF_HUB_OFFLINE=1 JLENS_MODEL=google/gemma-4-12B-it

scripts/stats_final.py

@@ -200,6 +200,45 @@ def main():
         lines.append(f"- {name}: A-only {r['a_only']}, B-only {r['b_only']}, "
                      f"n={r['n']}, p={r['p']:.4g} ({sig})")
 
+    # ---------- 2b. token baselines, per-item (oracle vs deployable) ------
+    try:
+        d = json.load(open(OUT / "eval_bestof3_select.json"))
+        sel = {it["task_id"]: it["selected"] for it in d["per_item"]}
+        orc = {it["task_id"]: it["oracle"] for it in d["per_item"]}
+        d2 = json.load(open(OUT / "eval_bestof3_pi.json"))
+        orc_pi = {it["task_id"]: it["ok"] for it in d2["per_item"]}
+        lines += ["", "## Token baselines (paired, per-item)", ""]
+        for name, m in [("best-of-3 ORACLE (any-pass)", orc_pi),
+                        ("best-of-3 oracle (selector run)", orc),
+                        ("best-of-3 DEPLOYABLE (logprob-selected)", sel)]:
+            o, h = acc_ci(m), acc_ci(m, mbpp_hard)
+            lines.append(f"- **{name}**: overall {fmt(o)}; hard {fmt(h)}")
+            report[name] = {"overall": o, "hard": h}
+        for base_name, m in [("oracle", orc_pi), ("deployable", sel)]:
+            for arm_name, arm in [("loop k=4", loop4), ("distill k=1", dist1)]:
+                for sub, subn in [(None, "overall"), (mbpp_hard, "hard")]:
+                    r = mcnemar(pair(arm, m, sub))
+                    report["mcnemar"][f"bo3-{base_name} vs {arm_name}, {subn}"] = r
+                    sig = "**significant**" if r["p"] < 0.05 else "n.s."
+                    lines.append(f"- bo3-{base_name} vs {arm_name}, {subn}: "
+                                 f"arm-only {r['a_only']}, bo3-only "
+                                 f"{r['b_only']}, p={r['p']:.4g} ({sig})")
+        d3 = json.load(open(OUT / "eval_budgetcot_pi.json"))
+        bc = {it["task_id"]: it["ok"] for it in d3["per_item"]}
+        o, h = acc_ci(bc), acc_ci(bc, mbpp_hard)
+        lines.append(f"- **budget-CoT-50 (per-item rerun)**: overall {fmt(o)}; "
+                     f"hard {fmt(h)}")
+        for arm_name, arm in [("loop k=4", loop4), ("distill k=1", dist1)]:
+            for sub, subn in [(None, "overall"), (mbpp_hard, "hard")]:
+                r = mcnemar(pair(arm, bc, sub))
+                report["mcnemar"][f"budget-cot vs {arm_name}, {subn}"] = r
+                sig = "**significant**" if r["p"] < 0.05 else "n.s."
+                lines.append(f"- budget-CoT vs {arm_name}, {subn}: arm-only "
+                             f"{r['a_only']}, cot-only {r['b_only']}, "
+                             f"p={r['p']:.4g} ({sig})")
+    except FileNotFoundError as e:
+        lines.append(f"(token-baseline per-item files incomplete: {e})")
+
     # ---------- 3. pooled hard bucket across benchmarks ----------
     lines += ["", "## Pooled hard bucket (MBPP + HumanEval + Rust)",
               "", "Paired within-item k>0 vs k=0, counts pooled across "