budget-CoT per-item: paired ties with latent arms confirmed (p>=0.69)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

PAPER.md

@@ -247,7 +247,10 @@ samples, so the comparison is exact), best-of-3 is statistically
 indistinguishable from the latent arms overall, *behind* them
 directionally on the hard bucket, and pays ≈3× visible tokens and serial
 decode for it. Budget-CoT-50 remains the strongest honest token baseline
-(53.8% overall, hard 40.0%). The implant's advantages at matched compute
+(53.8% overall, hard 40.0%; per-item rerun 53.8/38.2) — and the paired
+tests confirm it is a *tie* with the latent arms on both axes (p≥0.69 vs
+loop and distill), at the price of 50 visible tokens and their serial
+decode latency. The implant's advantages at matched compute
 without a verifier: zero visible tokens, zero decode overhead, and the
 hard-slice crown under distillation (46%). Where a task *does* come with a
 cheap verifier, oracle-style sampling is the better overall-accuracy

results-loop/STATS.md

@@ -44,7 +44,11 @@
 - bo3-deployable vs loop k=4, hard: arm-only 16, bo3-only 7, p=0.09314 (n.s.)
 - bo3-deployable vs distill k=1, overall: arm-only 31, bo3-only 34, p=0.8043 (n.s.)
 - bo3-deployable vs distill k=1, hard: arm-only 15, bo3-only 7, p=0.1338 (n.s.)
-(token-baseline per-item files incomplete: [Errno 2] No such file or directory: '/home/nils/jspace/results-loop/eval_budgetcot_pi.json')
+- **budget-CoT-50 (per-item rerun)**: overall 0.538 [0.494, 0.581] (n=500); hard 0.382 [0.265, 0.514] (n=55)
+- budget-CoT vs loop k=4, overall: arm-only 36, cot-only 37, p=1 (n.s.)
+- budget-CoT vs loop k=4, hard: arm-only 14, cot-only 11, p=0.69 (n.s.)
+- budget-CoT vs distill k=1, overall: arm-only 29, cot-only 26, p=0.7877 (n.s.)
+- budget-CoT vs distill k=1, hard: arm-only 8, cot-only 6, p=0.7905 (n.s.)
 
 ## Pooled hard bucket (MBPP + HumanEval + Rust)
 

results-loop/stats_final.json

@@ -370,6 +370,30 @@
    "a_only": 15,
    "b_only": 7,
    "p": 0.13380050659179688
+  },
+  "budget-cot vs loop k=4, overall": {
+   "n": 500,
+   "a_only": 36,
+   "b_only": 37,
+   "p": 1.0
+  },
+  "budget-cot vs loop k=4, hard": {
+   "n": 55,
+   "a_only": 14,
+   "b_only": 11,
+   "p": 0.6900379657745361
+  },
+  "budget-cot vs distill k=1, overall": {
+   "n": 500,
+   "a_only": 29,
+   "b_only": 26,
+   "p": 0.7877061896700435
+  },
+  "budget-cot vs distill k=1, hard": {
+   "n": 55,
+   "a_only": 8,
+   "b_only": 6,
+   "p": 0.79052734375
   }
  },
  "best-of-3 ORACLE (any-pass)": {