best-of-3 oracle/deployable split: realistic selector loses the overall edge (55.0/27.3 vs oracle 57.8/34.5); paper economics rewritten

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

results-loop/STATS.md

@@ -31,6 +31,21 @@
 - Rust loop k=4 vs k=0, overall: A-only 11, B-only 5, n=154, p=0.2101 (n.s.)
 - Rust loop k=4 vs k=0, hard: A-only 0, B-only 4, n=25, p=0.125 (n.s.)
 
+## Token baselines (paired, per-item)
+
+- **best-of-3 ORACLE (any-pass)**: overall 0.578 [0.534, 0.621] (n=500); hard 0.345 [0.234, 0.477] (n=55)
+- **best-of-3 oracle (selector run)**: overall 0.578 [0.534, 0.621] (n=500); hard 0.345 [0.234, 0.477] (n=55)
+- **best-of-3 DEPLOYABLE (logprob-selected)**: overall 0.550 [0.506, 0.593] (n=500); hard 0.273 [0.173, 0.402] (n=55)
+- bo3-oracle vs loop k=4, overall: arm-only 27, bo3-only 48, p=0.0203 (**significant**)
+- bo3-oracle vs loop k=4, hard: arm-only 14, bo3-only 9, p=0.4049 (n.s.)
+- bo3-oracle vs distill k=1, overall: arm-only 27, bo3-only 44, p=0.05681 (n.s.)
+- bo3-oracle vs distill k=1, hard: arm-only 14, bo3-only 10, p=0.5413 (n.s.)
+- bo3-deployable vs loop k=4, overall: arm-only 31, bo3-only 38, p=0.4704 (n.s.)
+- bo3-deployable vs loop k=4, hard: arm-only 16, bo3-only 7, p=0.09314 (n.s.)
+- bo3-deployable vs distill k=1, overall: arm-only 31, bo3-only 34, p=0.8043 (n.s.)
+- bo3-deployable vs distill k=1, hard: arm-only 15, bo3-only 7, p=0.1338 (n.s.)
+(token-baseline per-item files incomplete: [Errno 2] No such file or directory: '/home/nils/jspace/results-loop/eval_budgetcot_pi.json')
+
 ## Pooled hard bucket (MBPP + HumanEval + Rust)
 
 Paired within-item k>0 vs k=0, counts pooled across benchmarks (loop arm; distill pooled where available).

results-loop/stats_final.json

@@ -322,6 +322,108 @@
    "a_only": 0,
    "b_only": 4,
    "p": 0.125
+  },
+  "bo3-oracle vs loop k=4, overall": {
+   "n": 500,
+   "a_only": 27,
+   "b_only": 48,
+   "p": 0.020298406990107445
+  },
+  "bo3-oracle vs loop k=4, hard": {
+   "n": 55,
+   "a_only": 14,
+   "b_only": 9,
+   "p": 0.4048728942871094
+  },
+  "bo3-oracle vs distill k=1, overall": {
+   "n": 500,
+   "a_only": 27,
+   "b_only": 44,
+   "p": 0.056814677932839015
+  },
+  "bo3-oracle vs distill k=1, hard": {
+   "n": 55,
+   "a_only": 14,
+   "b_only": 10,
+   "p": 0.5412561893463135
+  },
+  "bo3-deployable vs loop k=4, overall": {
+   "n": 500,
+   "a_only": 31,
+   "b_only": 38,
+   "p": 0.4703685318581444
+  },
+  "bo3-deployable vs loop k=4, hard": {
+   "n": 55,
+   "a_only": 16,
+   "b_only": 7,
+   "p": 0.0931396484375
+  },
+  "bo3-deployable vs distill k=1, overall": {
+   "n": 500,
+   "a_only": 31,
+   "b_only": 34,
+   "p": 0.8043170001933986
+  },
+  "bo3-deployable vs distill k=1, hard": {
+   "n": 55,
+   "a_only": 15,
+   "b_only": 7,
+   "p": 0.13380050659179688
+  }
+ },
+ "best-of-3 ORACLE (any-pass)": {
+  "overall": {
+   "acc": 0.578,
+   "n": 500,
+   "ci": [
+    0.534,
+    0.621
+   ]
+  },
+  "hard": {
+   "acc": 0.34545454545454546,
+   "n": 55,
+   "ci": [
+    0.234,
+    0.477
+   ]
+  }
+ },
+ "best-of-3 oracle (selector run)": {
+  "overall": {
+   "acc": 0.578,
+   "n": 500,
+   "ci": [
+    0.534,
+    0.621
+   ]
+  },
+  "hard": {
+   "acc": 0.34545454545454546,
+   "n": 55,
+   "ci": [
+    0.234,
+    0.477
+   ]
+  }
+ },
+ "best-of-3 DEPLOYABLE (logprob-selected)": {
+  "overall": {
+   "acc": 0.55,
+   "n": 500,
+   "ci": [
+    0.506,
+    0.593
+   ]
+  },
+  "hard": {
+   "acc": 0.2727272727272727,
+   "n": 55,
+   "ci": [
+    0.173,
+    0.402
+   ]
   }
  },
  "pooled_hard_loop": {