nils/jspace
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

budget-CoT per-item: paired ties with latent arms confirmed (p>=0.69)

Browse Source 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Nils
2026-07-14 17:29:56 +02:00
co-authored by Claude Fable 5
parent 2ce646d175
commit caef95f10f
4 changed files with 2043 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
results-loop/STATS.md
+5 -1
View File
@@ -44,7 +44,11 @@
- bo3-deployable vs loop k=4, hard: arm-only 16, bo3-only 7, p=0.09314 (n.s.)
- bo3-deployable vs distill k=1, overall: arm-only 31, bo3-only 34, p=0.8043 (n.s.)
- bo3-deployable vs distill k=1, hard: arm-only 15, bo3-only 7, p=0.1338 (n.s.)
(token-baseline per-item files incomplete: [Errno 2] No such file or directory: '/home/nils/jspace/results-loop/eval_budgetcot_pi.json')
- **budget-CoT-50 (per-item rerun)**: overall 0.538 [0.494, 0.581] (n=500); hard 0.382 [0.265, 0.514] (n=55)
- budget-CoT vs loop k=4, overall: arm-only 36, cot-only 37, p=1 (n.s.)
- budget-CoT vs loop k=4, hard: arm-only 14, cot-only 11, p=0.69 (n.s.)
- budget-CoT vs distill k=1, overall: arm-only 29, cot-only 26, p=0.7877 (n.s.)
- budget-CoT vs distill k=1, hard: arm-only 8, cot-only 6, p=0.7905 (n.s.)
## Pooled hard bucket (MBPP + HumanEval + Rust)