CPU endgame: stats pass (Wilson/McNemar/pooled hard), final figures, PAPER.md rewrite around amortizable-content thesis
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Nils
Claude Fable 5
@@ -0,0 +1,47 @@
|
||||
# Final statistics pass
|
||||
|
||||
## Headline numbers (Wilson 95% CIs)
|
||||
|
||||
- **MBPP loop s0 k=0 (base)**: overall 0.518 [0.474, 0.561] (n=500); hard 0.055 [0.019, 0.149] (n=55)
|
||||
- **MBPP loop s0 k=2**: overall 0.530 [0.486, 0.573] (n=500); hard 0.309 [0.203, 0.440] (n=55)
|
||||
- **MBPP loop s0 k=4**: overall 0.536 [0.492, 0.579] (n=500); hard 0.436 [0.314, 0.567] (n=55)
|
||||
- **MBPP distill s1 k=1 (FF)**: overall 0.544 [0.500, 0.587] (n=500); hard 0.418 [0.297, 0.550] (n=55)
|
||||
- **MBPP pause16 k=1**: overall 0.552 [0.508, 0.595] (n=500); hard 0.364 [0.249, 0.496] (n=55)
|
||||
- **MBPP stack-train k=4**: overall 0.524 [0.480, 0.567] (n=500); hard 0.345 [0.234, 0.477] (n=55)
|
||||
- **MBPP distill-in-loopmode k=2**: overall 0.458 [0.415, 0.502] (n=500); hard 0.200 [0.116, 0.324] (n=55)
|
||||
- **Rust transfer k=0**: overall 0.591 [0.512, 0.665] (n=154); hard 0.080 [0.022, 0.250] (n=25)
|
||||
- **Rust transfer k=4**: overall 0.552 [0.473, 0.628] (n=154); hard 0.240 [0.115, 0.434] (n=25)
|
||||
- **HumanEval loop k=4**: overall 0.665 [0.589, 0.732] (n=164); hard 0.316 [0.191, 0.475] (n=38)
|
||||
- **HumanEval distill k=1**: overall 0.646 [0.571, 0.715] (n=164); hard 0.237 [0.130, 0.392] (n=38)
|
||||
- **MBPP best-of-3 (compute-matched)**: overall 0.572 [0.528, 0.615] (n=500); hard 0.327 [0.218, 0.459] (n=55)
|
||||
- **MBPP budget-CoT-50**: overall 0.538 [0.494, 0.581] (n=500); hard 0.400 [0.281, 0.532] (n=55)
|
||||
- **MBPP distill, 8 runs (hard)**: mean 0.457 ± 0.046 sd (range 0.400-0.545); overall mean 0.555
|
||||
- **MBPP loop seeds k=4 (hard)**: mean 0.375 ± 0.055 sd (n_seeds=5)
|
||||
|
||||
## McNemar exact tests (paired on items)
|
||||
|
||||
- loop k=4 vs k=0, overall: A-only 30, B-only 39, n=500, p=0.3356 (n.s.)
|
||||
- loop k=4 vs k=0, hard: A-only 1, B-only 22, n=55, p=5.722e-06 (**significant**)
|
||||
- distill k=1 vs loop k=4, overall: A-only 33, B-only 37, n=500, p=0.7202 (n.s.)
|
||||
- distill k=1 vs loop k=4, hard: A-only 10, B-only 9, n=55, p=1 (n.s.)
|
||||
- stack-train k=4 vs distill k=1, hard: A-only 10, B-only 6, n=55, p=0.4545 (n.s.)
|
||||
- HumanEval loop k=4 vs k=0, overall: A-only 5, B-only 18, n=164, p=0.01062 (**significant**)
|
||||
- HumanEval distill k=1 vs k=0, overall: A-only 7, B-only 17, n=164, p=0.06391 (n.s.)
|
||||
- Rust loop k=4 vs k=0, overall: A-only 11, B-only 5, n=154, p=0.2101 (n.s.)
|
||||
- Rust loop k=4 vs k=0, hard: A-only 0, B-only 4, n=25, p=0.125 (n.s.)
|
||||
|
||||
## Pooled hard bucket (MBPP + HumanEval + Rust)
|
||||
|
||||
Paired within-item k>0 vs k=0, counts pooled across benchmarks (loop arm; distill pooled where available).
|
||||
|
||||
- **loop**: base 5/118 -> loop 42/118 (0.042 -> 0.356, CI [0.275, 0.446]), McNemar p=1.46e-10
|
||||
- **distill**: base 3/93 -> distill 32/93 (0.032 -> 0.344, CI [0.255, 0.445]), McNemar p=2.98e-08
|
||||
|
||||
## Label robustness (consensus-k0 hard set)
|
||||
|
||||
Hard bucket redefined as: labeled hard AND k=0 fails in every seed's own eval run (removes single-greedy-run selection noise).
|
||||
|
||||
- consensus hard set: 52 of 55 labeled-hard items
|
||||
- MBPP loop s0 k=4: labeled-hard 0.436 -> consensus-hard 0.423 [0.299, 0.558] (n=52)
|
||||
- MBPP distill s1 k=1 (FF): labeled-hard 0.418 -> consensus-hard 0.404 [0.282, 0.539] (n=52)
|
||||
- MBPP stack-train k=4: labeled-hard 0.345 -> consensus-hard 0.346 [0.232, 0.482] (n=52)
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,26 @@
|
||||
{
|
||||
"0": {
|
||||
"acc": 0.5,
|
||||
"by_label": {
|
||||
"easy": 0.9672131147540983,
|
||||
"hard": 0.17857142857142858,
|
||||
"drop": 0.02
|
||||
}
|
||||
},
|
||||
"2": {
|
||||
"acc": 0.504,
|
||||
"by_label": {
|
||||
"easy": 0.9098360655737705,
|
||||
"hard": 0.35714285714285715,
|
||||
"drop": 0.05
|
||||
}
|
||||
},
|
||||
"4": {
|
||||
"acc": 0.512,
|
||||
"by_label": {
|
||||
"easy": 0.9098360655737705,
|
||||
"hard": 0.42857142857142855,
|
||||
"drop": 0.05
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,26 @@
|
||||
{
|
||||
"0": {
|
||||
"acc": 0.496,
|
||||
"by_label": {
|
||||
"easy": 0.9672131147540983,
|
||||
"hard": 0.10714285714285714,
|
||||
"drop": 0.03
|
||||
}
|
||||
},
|
||||
"2": {
|
||||
"acc": 0.516,
|
||||
"by_label": {
|
||||
"easy": 0.9344262295081968,
|
||||
"hard": 0.39285714285714285,
|
||||
"drop": 0.04
|
||||
}
|
||||
},
|
||||
"4": {
|
||||
"acc": 0.524,
|
||||
"by_label": {
|
||||
"easy": 0.9098360655737705,
|
||||
"hard": 0.4642857142857143,
|
||||
"drop": 0.07
|
||||
}
|
||||
}
|
||||
}
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 84 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 66 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 80 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 36 KiB |
@@ -0,0 +1,369 @@
|
||||
{
|
||||
"MBPP loop s0 k=0 (base)": {
|
||||
"overall": {
|
||||
"acc": 0.518,
|
||||
"n": 500,
|
||||
"ci": [
|
||||
0.474,
|
||||
0.561
|
||||
]
|
||||
},
|
||||
"hard": {
|
||||
"acc": 0.05454545454545454,
|
||||
"n": 55,
|
||||
"ci": [
|
||||
0.019,
|
||||
0.149
|
||||
]
|
||||
}
|
||||
},
|
||||
"MBPP loop s0 k=2": {
|
||||
"overall": {
|
||||
"acc": 0.53,
|
||||
"n": 500,
|
||||
"ci": [
|
||||
0.486,
|
||||
0.573
|
||||
]
|
||||
},
|
||||
"hard": {
|
||||
"acc": 0.3090909090909091,
|
||||
"n": 55,
|
||||
"ci": [
|
||||
0.203,
|
||||
0.44
|
||||
]
|
||||
}
|
||||
},
|
||||
"MBPP loop s0 k=4": {
|
||||
"overall": {
|
||||
"acc": 0.536,
|
||||
"n": 500,
|
||||
"ci": [
|
||||
0.492,
|
||||
0.579
|
||||
]
|
||||
},
|
||||
"hard": {
|
||||
"acc": 0.43636363636363634,
|
||||
"n": 55,
|
||||
"ci": [
|
||||
0.314,
|
||||
0.567
|
||||
]
|
||||
}
|
||||
},
|
||||
"MBPP distill s1 k=1 (FF)": {
|
||||
"overall": {
|
||||
"acc": 0.544,
|
||||
"n": 500,
|
||||
"ci": [
|
||||
0.5,
|
||||
0.587
|
||||
]
|
||||
},
|
||||
"hard": {
|
||||
"acc": 0.41818181818181815,
|
||||
"n": 55,
|
||||
"ci": [
|
||||
0.297,
|
||||
0.55
|
||||
]
|
||||
}
|
||||
},
|
||||
"MBPP pause16 k=1": {
|
||||
"overall": {
|
||||
"acc": 0.552,
|
||||
"n": 500,
|
||||
"ci": [
|
||||
0.508,
|
||||
0.595
|
||||
]
|
||||
},
|
||||
"hard": {
|
||||
"acc": 0.36363636363636365,
|
||||
"n": 55,
|
||||
"ci": [
|
||||
0.249,
|
||||
0.496
|
||||
]
|
||||
}
|
||||
},
|
||||
"MBPP stack-train k=4": {
|
||||
"overall": {
|
||||
"acc": 0.524,
|
||||
"n": 500,
|
||||
"ci": [
|
||||
0.48,
|
||||
0.567
|
||||
]
|
||||
},
|
||||
"hard": {
|
||||
"acc": 0.34545454545454546,
|
||||
"n": 55,
|
||||
"ci": [
|
||||
0.234,
|
||||
0.477
|
||||
]
|
||||
}
|
||||
},
|
||||
"MBPP distill-in-loopmode k=2": {
|
||||
"overall": {
|
||||
"acc": 0.458,
|
||||
"n": 500,
|
||||
"ci": [
|
||||
0.415,
|
||||
0.502
|
||||
]
|
||||
},
|
||||
"hard": {
|
||||
"acc": 0.2,
|
||||
"n": 55,
|
||||
"ci": [
|
||||
0.116,
|
||||
0.324
|
||||
]
|
||||
}
|
||||
},
|
||||
"Rust transfer k=0": {
|
||||
"overall": {
|
||||
"acc": 0.5909090909090909,
|
||||
"n": 154,
|
||||
"ci": [
|
||||
0.512,
|
||||
0.665
|
||||
]
|
||||
},
|
||||
"hard": {
|
||||
"acc": 0.08,
|
||||
"n": 25,
|
||||
"ci": [
|
||||
0.022,
|
||||
0.25
|
||||
]
|
||||
}
|
||||
},
|
||||
"Rust transfer k=4": {
|
||||
"overall": {
|
||||
"acc": 0.551948051948052,
|
||||
"n": 154,
|
||||
"ci": [
|
||||
0.473,
|
||||
0.628
|
||||
]
|
||||
},
|
||||
"hard": {
|
||||
"acc": 0.24,
|
||||
"n": 25,
|
||||
"ci": [
|
||||
0.115,
|
||||
0.434
|
||||
]
|
||||
}
|
||||
},
|
||||
"HumanEval loop k=4": {
|
||||
"overall": {
|
||||
"acc": 0.6646341463414634,
|
||||
"n": 164,
|
||||
"ci": [
|
||||
0.589,
|
||||
0.732
|
||||
]
|
||||
},
|
||||
"hard": {
|
||||
"acc": 0.3157894736842105,
|
||||
"n": 38,
|
||||
"ci": [
|
||||
0.191,
|
||||
0.475
|
||||
]
|
||||
}
|
||||
},
|
||||
"HumanEval distill k=1": {
|
||||
"overall": {
|
||||
"acc": 0.6463414634146342,
|
||||
"n": 164,
|
||||
"ci": [
|
||||
0.571,
|
||||
0.715
|
||||
]
|
||||
},
|
||||
"hard": {
|
||||
"acc": 0.23684210526315788,
|
||||
"n": 38,
|
||||
"ci": [
|
||||
0.13,
|
||||
0.392
|
||||
]
|
||||
}
|
||||
},
|
||||
"MBPP best-of-3 (compute-matched)": {
|
||||
"overall": {
|
||||
"acc": 0.572,
|
||||
"n": 500,
|
||||
"ci": [
|
||||
0.528,
|
||||
0.615
|
||||
]
|
||||
},
|
||||
"hard": {
|
||||
"acc": 0.32727272727272727,
|
||||
"n": 55,
|
||||
"ci": [
|
||||
0.218,
|
||||
0.459
|
||||
]
|
||||
}
|
||||
},
|
||||
"MBPP budget-CoT-50": {
|
||||
"overall": {
|
||||
"acc": 0.538,
|
||||
"n": 500,
|
||||
"ci": [
|
||||
0.494,
|
||||
0.581
|
||||
]
|
||||
},
|
||||
"hard": {
|
||||
"acc": 0.4,
|
||||
"n": 55,
|
||||
"ci": [
|
||||
0.281,
|
||||
0.532
|
||||
]
|
||||
}
|
||||
},
|
||||
"distill_seed_spread": {
|
||||
"hard": [
|
||||
0.4909090909090909,
|
||||
0.41818181818181815,
|
||||
0.5454545454545454,
|
||||
0.43636363636363634,
|
||||
0.45454545454545453,
|
||||
0.4727272727272727,
|
||||
0.43636363636363634,
|
||||
0.4
|
||||
],
|
||||
"overall": [
|
||||
0.55,
|
||||
0.544,
|
||||
0.562,
|
||||
0.546,
|
||||
0.566,
|
||||
0.562,
|
||||
0.564,
|
||||
0.55
|
||||
]
|
||||
},
|
||||
"loop_seed_spread_hard": [
|
||||
0.43636363636363634,
|
||||
0.41818181818181815,
|
||||
0.3090909090909091,
|
||||
0.32727272727272727,
|
||||
0.38181818181818183
|
||||
],
|
||||
"mcnemar": {
|
||||
"loop k=4 vs k=0, overall": {
|
||||
"n": 500,
|
||||
"a_only": 30,
|
||||
"b_only": 39,
|
||||
"p": 0.33555761823401514
|
||||
},
|
||||
"loop k=4 vs k=0, hard": {
|
||||
"n": 55,
|
||||
"a_only": 1,
|
||||
"b_only": 22,
|
||||
"p": 5.7220458984375e-06
|
||||
},
|
||||
"distill k=1 vs loop k=4, overall": {
|
||||
"n": 500,
|
||||
"a_only": 33,
|
||||
"b_only": 37,
|
||||
"p": 0.7202027723528613
|
||||
},
|
||||
"distill k=1 vs loop k=4, hard": {
|
||||
"n": 55,
|
||||
"a_only": 10,
|
||||
"b_only": 9,
|
||||
"p": 1.0
|
||||
},
|
||||
"stack-train k=4 vs distill k=1, hard": {
|
||||
"n": 55,
|
||||
"a_only": 10,
|
||||
"b_only": 6,
|
||||
"p": 0.454498291015625
|
||||
},
|
||||
"HumanEval loop k=4 vs k=0, overall": {
|
||||
"n": 164,
|
||||
"a_only": 5,
|
||||
"b_only": 18,
|
||||
"p": 0.010622024536132812
|
||||
},
|
||||
"HumanEval distill k=1 vs k=0, overall": {
|
||||
"n": 164,
|
||||
"a_only": 7,
|
||||
"b_only": 17,
|
||||
"p": 0.06391465663909912
|
||||
},
|
||||
"Rust loop k=4 vs k=0, overall": {
|
||||
"n": 154,
|
||||
"a_only": 11,
|
||||
"b_only": 5,
|
||||
"p": 0.210113525390625
|
||||
},
|
||||
"Rust loop k=4 vs k=0, hard": {
|
||||
"n": 25,
|
||||
"a_only": 0,
|
||||
"b_only": 4,
|
||||
"p": 0.125
|
||||
}
|
||||
},
|
||||
"pooled_hard_loop": {
|
||||
"base": 5,
|
||||
"arm": 42,
|
||||
"n": 118,
|
||||
"mcnemar": {
|
||||
"n": 118,
|
||||
"a_only": 1,
|
||||
"b_only": 38,
|
||||
"p": 1.4551915228366852e-10
|
||||
}
|
||||
},
|
||||
"pooled_hard_distill": {
|
||||
"base": 3,
|
||||
"arm": 32,
|
||||
"n": 93,
|
||||
"mcnemar": {
|
||||
"n": 93,
|
||||
"a_only": 1,
|
||||
"b_only": 30,
|
||||
"p": 2.9802322387695312e-08
|
||||
}
|
||||
},
|
||||
"consensus_hard": {
|
||||
"MBPP loop s0 k=4": {
|
||||
"acc": 0.4230769230769231,
|
||||
"n": 52,
|
||||
"ci": [
|
||||
0.299,
|
||||
0.558
|
||||
]
|
||||
},
|
||||
"MBPP distill s1 k=1 (FF)": {
|
||||
"acc": 0.40384615384615385,
|
||||
"n": 52,
|
||||
"ci": [
|
||||
0.282,
|
||||
0.539
|
||||
]
|
||||
},
|
||||
"MBPP stack-train k=4": {
|
||||
"acc": 0.34615384615384615,
|
||||
"n": 52,
|
||||
"ci": [
|
||||
0.232,
|
||||
0.482
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user