item 17 scored: GSM-only current recipe fails as predicted; GSM chapter closed

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Nils
2026-07-15 21:14:58 +02:00
co-authored by Claude Fable 5
parent 2cd1f37f5f
commit cb0f1d950c
2 changed files with 4157 additions and 0 deletions
+14
View File
@@ -345,3 +345,17 @@ levels — only seed means are.
(noise-s0, h2048) variants have no mechanism by which task change
could invert their MBPP verdict. Only the recipe family's best member
(this item) is informative on GSM8K.
--- Outcome, item 17 (GSM-only, current recipe, scored 2026-07-15):
predictions (a) and (b) CONFIRMED, (c) nominally exceeded but not
meaningfully. hard 8.7/5.5/4.7% at k=1/2/4 (below the 10% bar; nowhere
near the 15% reopen threshold); easy 93.1 -> 48-52% at k>0; overall
11.7/10.9/10.2 vs k0's 10.5 — the k=1 cell is +1.2 points nominal
(~3 items at n=256, not significant), the rest below. Removing the
mixed-task interference bought ~2 points over adapter_uni (9.4 -> 11.7
at k=1) — interference was real but marginal, not masking a capability.
The GSM8K chapter is closed: the recipe family's best member, trained
on GSM alone in the correct regime, delivers no usable gain and the
standard fidelity damage; combined with the scope note, the boundary
claim (structural: supervision density + state-evolution bottleneck)
is fully supported.