nils/jspace
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

n=500 controls: paired net-effect significant (loop vs untrained merge 17-4, p=0.007); ladder and net accounting finalized

Browse Source 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Nils
2026-07-14 20:08:57 +02:00
co-authored by Claude Fable 5
parent 1d4cffd6ad
commit 0fd93cb328
8 changed files with 10160 additions and 19 deletions
Download Patch File Download Diff File Expand all files Collapse all files
results-loop/STATS.md
+6
View File
@@ -2,6 +2,8 @@
## Headline numbers (Wilson 95% CIs)
- **MBPP untrained merge k=4 (n=500 rerun)**: overall 0.502 [0.458, 0.546] (n=500); hard 0.200 [0.116, 0.324] (n=55)
- **MBPP trained FF k=1 (n=500 rerun)**: overall 0.536 [0.492, 0.579] (n=500); hard 0.273 [0.173, 0.402] (n=55)
- **MBPP loop s0 k=0 (base)**: overall 0.518 [0.474, 0.561] (n=500); hard 0.055 [0.019, 0.149] (n=55)
- **MBPP loop s0 k=2**: overall 0.530 [0.486, 0.573] (n=500); hard 0.309 [0.203, 0.440] (n=55)
- **MBPP loop s0 k=4**: overall 0.536 [0.492, 0.579] (n=500); hard 0.436 [0.314, 0.567] (n=55)
@@ -22,6 +24,10 @@
- loop k=4 vs k=0, overall: A-only 30, B-only 39, n=500, p=0.3356 (n.s.)
- loop k=4 vs k=0, hard: A-only 1, B-only 22, n=55, p=5.722e-06 (**significant**)
- loop k=4 vs UNTRAINED merge k=4, hard (net effect): A-only 4, B-only 17, n=55, p=0.007197 (**significant**)
- loop k=4 vs UNTRAINED merge k=4, overall: A-only 29, B-only 46, n=500, p=0.06395 (n.s.)
- trained FF vs UNTRAINED merge, hard: A-only 7, B-only 11, n=55, p=0.4807 (n.s.)
- distill k=1 vs UNTRAINED merge k=4, hard: A-only 3, B-only 15, n=55, p=0.007538 (**significant**)
- distill k=1 vs loop k=4, overall: A-only 33, B-only 37, n=500, p=0.7202 (n.s.)
- distill k=1 vs loop k=4, hard: A-only 10, B-only 9, n=55, p=1 (n.s.)
- stack-train k=4 vs distill k=1, hard: A-only 10, B-only 6, n=55, p=0.4545 (n.s.)