budget-CoT per-item: paired ties with latent arms confirmed (p>=0.69)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

results-loop/STATS.md

@@ -44,7 +44,11 @@
 - bo3-deployable vs loop k=4, hard: arm-only 16, bo3-only 7, p=0.09314 (n.s.)
 - bo3-deployable vs distill k=1, overall: arm-only 31, bo3-only 34, p=0.8043 (n.s.)
 - bo3-deployable vs distill k=1, hard: arm-only 15, bo3-only 7, p=0.1338 (n.s.)
-(token-baseline per-item files incomplete: [Errno 2] No such file or directory: '/home/nils/jspace/results-loop/eval_budgetcot_pi.json')
+- **budget-CoT-50 (per-item rerun)**: overall 0.538 [0.494, 0.581] (n=500); hard 0.382 [0.265, 0.514] (n=55)
+- budget-CoT vs loop k=4, overall: arm-only 36, cot-only 37, p=1 (n.s.)
+- budget-CoT vs loop k=4, hard: arm-only 14, cot-only 11, p=0.69 (n.s.)
+- budget-CoT vs distill k=1, overall: arm-only 29, cot-only 26, p=0.7877 (n.s.)
+- budget-CoT vs distill k=1, hard: arm-only 8, cot-only 6, p=0.7905 (n.s.)
 
 ## Pooled hard bucket (MBPP + HumanEval + Rust)
 

results-loop/stats_final.json

@@ -370,6 +370,30 @@
    "a_only": 15,
    "b_only": 7,
    "p": 0.13380050659179688
+  },
+  "budget-cot vs loop k=4, overall": {
+   "n": 500,
+   "a_only": 36,
+   "b_only": 37,
+   "p": 1.0
+  },
+  "budget-cot vs loop k=4, hard": {
+   "n": 55,
+   "a_only": 14,
+   "b_only": 11,
+   "p": 0.6900379657745361
+  },
+  "budget-cot vs distill k=1, overall": {
+   "n": 500,
+   "a_only": 29,
+   "b_only": 26,
+   "p": 0.7877061896700435
+  },
+  "budget-cot vs distill k=1, hard": {
+   "n": 55,
+   "a_only": 8,
+   "b_only": 6,
+   "p": 0.79052734375
   }
  },
  "best-of-3 ORACLE (any-pass)": {