best-of-3 oracle/deployable split: realistic selector loses the overall edge (55.0/27.3 vs oracle 57.8/34.5); paper economics rewritten

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

PAPER.md

@@ -31,7 +31,8 @@ A complete attribution program then deflates the mechanism's mystique: the
 loop's content is *amortizable* (plan-distillation into the same adapter,
 recurrence-free, matches it; nothing stacks; looping distilled weights
 degrades them), width rivals depth (trained pause registers reach 36.4%),
-and compute-matched token baselines win overall accuracy outright. What
+and verifier-assisted (oracle) sampling wins overall accuracy at matched
+compute — though the *deployable* selector loses that edge entirely. What
 survives is precise: the implant owns exactly the plan-dependent slice at
 zero visible tokens and zero decode cost, transfers with the substrate
 rather than the task, and its placement is dictated by the lens. At 12B a
@@ -88,11 +89,14 @@ not laws.)*
    (parallel, prefill-shaped), zero decode overhead, bit-exact KV-cache
    write-in, k=0 recovers the base model exactly. Its competition at
    comparable compute (accounting in Appendix A — FLOPs, wall-clock, and
-   token budget do not rank the arms the same way): best-of-3 sampling wins
-   overall accuracy outright (57.2%); a
-   50-token visible plan ties the hard bucket. The value proposition is
-   *only*: no visible tokens, no decode latency, and the hard-slice
-   specialization (distill's 46% > budget-CoT's 40% > best-of-3's 33%).
+   token budget do not rank the arms the same way): *oracle* best-of-3
+   (verifier-assisted) wins overall accuracy (57.8%, paired p=0.020 vs
+   loop), but the *deployable* logprob-selected variant drops to 55.0%
+   overall / 27.3% hard — indistinguishable from the latent arms overall
+   and directionally behind on hard. A 50-token visible plan ties the hard
+   bucket. The value proposition without a verifier: no visible tokens, no
+   decode latency, and the hard-slice specialization (distill's 46% >
+   budget-CoT's 40% > deployable sampling's 27%).
 
 6. **Scale transfers only with a state-dependent stability dial.** At 12B the
    2B-tuned constant α=0.3 collapses overall accuracy (72.6%→43.0%); the
@@ -228,20 +232,26 @@ distinctive value is that it finds this content *without* plan supervision
 ### 3.4 Compute-matched honesty
 
 At approximately matched FLOPs (Appendix A gives the accounting, separated
-into FLOPs, wall-clock, and token budget), token-space baselines are strong:
-best-of-3 sampling wins overall accuracy against every latent arm (57.2%,
-CI [52.8, 61.5], vs loop 53.6 [49.2, 57.9] — point estimate higher, CIs
-overlap) by preserving easy items perfectly while sampling rescues some hard
-ones. A 50-token visible plan ties the loop's hard bucket. Both baselines
-are being rerun with per-item logs to enable paired tests against the latent
-arms; until those land, the overall-accuracy comparison rests on overlapping
-CIs and is stated as point-estimate-level. [BASELINES-PI PENDING]
-The latent implant's surviving advantages are qualitative: zero visible
-tokens (silent), zero decode overhead (prefill-parallel; sampling and CoT
-pay serially at bandwidth-bound decode), and the hard-slice crown under
-distillation (46% vs 40% budget-CoT vs 33% best-of-3). For deployment this
-means: the implant is a *latency/token-budget* technology with a side
-specialization in plan-dependent items — not an accuracy technology.
+into FLOPs, wall-clock, and token budget), the token-space comparison
+splits into two very different claims:
+
+| best-of-3 variant | overall | hard | vs loop (paired) |
+|---|---|---|---|
+| **oracle** (any-of-3 passes; needs a perfect verifier) | 57.8% | 34.5% | beats loop overall, p=0.020 |
+| **deployable** (highest mean logprob of 3) | 55.0% | 27.3% | n.s. overall (p=0.47); loop ahead on hard 16–7 (p=0.09) |
+
+The earlier draft's "sampling wins overall" was the **oracle** number — an
+upper bound requiring an external verifier that MBPP's own tests provide
+but a deployment does not. With the deployable selector (identical seeded
+samples, so the comparison is exact), best-of-3 is statistically
+indistinguishable from the latent arms overall, *behind* them
+directionally on the hard bucket, and pays ≈3× visible tokens and serial
+decode for it. Budget-CoT-50 remains the strongest honest token baseline
+(53.8% overall, hard 40.0%). The implant's advantages at matched compute
+without a verifier: zero visible tokens, zero decode overhead, and the
+hard-slice crown under distillation (46%). Where a task *does* come with a
+cheap verifier, oracle-style sampling is the better overall-accuracy
+spend — both halves belong in the deployment picture.
 
 ### 3.5 Width vs depth, and the task boundary
 

results-loop/STATS.md

@@ -31,6 +31,21 @@
 - Rust loop k=4 vs k=0, overall: A-only 11, B-only 5, n=154, p=0.2101 (n.s.)
 - Rust loop k=4 vs k=0, hard: A-only 0, B-only 4, n=25, p=0.125 (n.s.)
 
+## Token baselines (paired, per-item)
+
+- **best-of-3 ORACLE (any-pass)**: overall 0.578 [0.534, 0.621] (n=500); hard 0.345 [0.234, 0.477] (n=55)
+- **best-of-3 oracle (selector run)**: overall 0.578 [0.534, 0.621] (n=500); hard 0.345 [0.234, 0.477] (n=55)
+- **best-of-3 DEPLOYABLE (logprob-selected)**: overall 0.550 [0.506, 0.593] (n=500); hard 0.273 [0.173, 0.402] (n=55)
+- bo3-oracle vs loop k=4, overall: arm-only 27, bo3-only 48, p=0.0203 (**significant**)
+- bo3-oracle vs loop k=4, hard: arm-only 14, bo3-only 9, p=0.4049 (n.s.)
+- bo3-oracle vs distill k=1, overall: arm-only 27, bo3-only 44, p=0.05681 (n.s.)
+- bo3-oracle vs distill k=1, hard: arm-only 14, bo3-only 10, p=0.5413 (n.s.)
+- bo3-deployable vs loop k=4, overall: arm-only 31, bo3-only 38, p=0.4704 (n.s.)
+- bo3-deployable vs loop k=4, hard: arm-only 16, bo3-only 7, p=0.09314 (n.s.)
+- bo3-deployable vs distill k=1, overall: arm-only 31, bo3-only 34, p=0.8043 (n.s.)
+- bo3-deployable vs distill k=1, hard: arm-only 15, bo3-only 7, p=0.1338 (n.s.)
+(token-baseline per-item files incomplete: [Errno 2] No such file or directory: '/home/nils/jspace/results-loop/eval_budgetcot_pi.json')
+
 ## Pooled hard bucket (MBPP + HumanEval + Rust)
 
 Paired within-item k>0 vs k=0, counts pooled across benchmarks (loop arm; distill pooled where available).

results-loop/stats_final.json

@@ -322,6 +322,108 @@
    "a_only": 0,
    "b_only": 4,
    "p": 0.125
+  },
+  "bo3-oracle vs loop k=4, overall": {
+   "n": 500,
+   "a_only": 27,
+   "b_only": 48,
+   "p": 0.020298406990107445
+  },
+  "bo3-oracle vs loop k=4, hard": {
+   "n": 55,
+   "a_only": 14,
+   "b_only": 9,
+   "p": 0.4048728942871094
+  },
+  "bo3-oracle vs distill k=1, overall": {
+   "n": 500,
+   "a_only": 27,
+   "b_only": 44,
+   "p": 0.056814677932839015
+  },
+  "bo3-oracle vs distill k=1, hard": {
+   "n": 55,
+   "a_only": 14,
+   "b_only": 10,
+   "p": 0.5412561893463135
+  },
+  "bo3-deployable vs loop k=4, overall": {
+   "n": 500,
+   "a_only": 31,
+   "b_only": 38,
+   "p": 0.4703685318581444
+  },
+  "bo3-deployable vs loop k=4, hard": {
+   "n": 55,
+   "a_only": 16,
+   "b_only": 7,
+   "p": 0.0931396484375
+  },
+  "bo3-deployable vs distill k=1, overall": {
+   "n": 500,
+   "a_only": 31,
+   "b_only": 34,
+   "p": 0.8043170001933986
+  },
+  "bo3-deployable vs distill k=1, hard": {
+   "n": 55,
+   "a_only": 15,
+   "b_only": 7,
+   "p": 0.13380050659179688
+  }
+ },
+ "best-of-3 ORACLE (any-pass)": {
+  "overall": {
+   "acc": 0.578,
+   "n": 500,
+   "ci": [
+    0.534,
+    0.621
+   ]
+  },
+  "hard": {
+   "acc": 0.34545454545454546,
+   "n": 55,
+   "ci": [
+    0.234,
+    0.477
+   ]
+  }
+ },
+ "best-of-3 oracle (selector run)": {
+  "overall": {
+   "acc": 0.578,
+   "n": 500,
+   "ci": [
+    0.534,
+    0.621
+   ]
+  },
+  "hard": {
+   "acc": 0.34545454545454546,
+   "n": 55,
+   "ci": [
+    0.234,
+    0.477
+   ]
+  }
+ },
+ "best-of-3 DEPLOYABLE (logprob-selected)": {
+  "overall": {
+   "acc": 0.55,
+   "n": 500,
+   "ci": [
+    0.506,
+    0.593
+   ]
+  },
+  "hard": {
+   "acc": 0.2727272727272727,
+   "n": 55,
+   "ci": [
+    0.173,
+    0.402
+   ]
   }
  },
  "pooled_hard_loop": {

scripts/stats_final.py

@@ -200,6 +200,45 @@ def main():
         lines.append(f"- {name}: A-only {r['a_only']}, B-only {r['b_only']}, "
                      f"n={r['n']}, p={r['p']:.4g} ({sig})")
 
+    # ---------- 2b. token baselines, per-item (oracle vs deployable) ------
+    try:
+        d = json.load(open(OUT / "eval_bestof3_select.json"))
+        sel = {it["task_id"]: it["selected"] for it in d["per_item"]}
+        orc = {it["task_id"]: it["oracle"] for it in d["per_item"]}
+        d2 = json.load(open(OUT / "eval_bestof3_pi.json"))
+        orc_pi = {it["task_id"]: it["ok"] for it in d2["per_item"]}
+        lines += ["", "## Token baselines (paired, per-item)", ""]
+        for name, m in [("best-of-3 ORACLE (any-pass)", orc_pi),
+                        ("best-of-3 oracle (selector run)", orc),
+                        ("best-of-3 DEPLOYABLE (logprob-selected)", sel)]:
+            o, h = acc_ci(m), acc_ci(m, mbpp_hard)
+            lines.append(f"- **{name}**: overall {fmt(o)}; hard {fmt(h)}")
+            report[name] = {"overall": o, "hard": h}
+        for base_name, m in [("oracle", orc_pi), ("deployable", sel)]:
+            for arm_name, arm in [("loop k=4", loop4), ("distill k=1", dist1)]:
+                for sub, subn in [(None, "overall"), (mbpp_hard, "hard")]:
+                    r = mcnemar(pair(arm, m, sub))
+                    report["mcnemar"][f"bo3-{base_name} vs {arm_name}, {subn}"] = r
+                    sig = "**significant**" if r["p"] < 0.05 else "n.s."
+                    lines.append(f"- bo3-{base_name} vs {arm_name}, {subn}: "
+                                 f"arm-only {r['a_only']}, bo3-only "
+                                 f"{r['b_only']}, p={r['p']:.4g} ({sig})")
+        d3 = json.load(open(OUT / "eval_budgetcot_pi.json"))
+        bc = {it["task_id"]: it["ok"] for it in d3["per_item"]}
+        o, h = acc_ci(bc), acc_ci(bc, mbpp_hard)
+        lines.append(f"- **budget-CoT-50 (per-item rerun)**: overall {fmt(o)}; "
+                     f"hard {fmt(h)}")
+        for arm_name, arm in [("loop k=4", loop4), ("distill k=1", dist1)]:
+            for sub, subn in [(None, "overall"), (mbpp_hard, "hard")]:
+                r = mcnemar(pair(arm, bc, sub))
+                report["mcnemar"][f"budget-cot vs {arm_name}, {subn}"] = r
+                sig = "**significant**" if r["p"] < 0.05 else "n.s."
+                lines.append(f"- budget-CoT vs {arm_name}, {subn}: arm-only "
+                             f"{r['a_only']}, cot-only {r['b_only']}, "
+                             f"p={r['p']:.4g} ({sig})")
+    except FileNotFoundError as e:
+        lines.append(f"(token-baseline per-item files incomplete: {e})")
+
     # ---------- 3. pooled hard bucket across benchmarks ----------
     lines += ["", "## Pooled hard bucket (MBPP + HumanEval + Rust)",
               "", "Paired within-item k>0 vs k=0, counts pooled across "