item 17 scored: GSM-only current recipe fails as predicted; GSM chapter closed
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -345,3 +345,17 @@ levels — only seed means are.
|
||||
(noise-s0, h2048) variants have no mechanism by which task change
|
||||
could invert their MBPP verdict. Only the recipe family's best member
|
||||
(this item) is informative on GSM8K.
|
||||
|
||||
--- Outcome, item 17 (GSM-only, current recipe, scored 2026-07-15):
|
||||
predictions (a) and (b) CONFIRMED, (c) nominally exceeded but not
|
||||
meaningfully. hard 8.7/5.5/4.7% at k=1/2/4 (below the 10% bar; nowhere
|
||||
near the 15% reopen threshold); easy 93.1 -> 48-52% at k>0; overall
|
||||
11.7/10.9/10.2 vs k0's 10.5 — the k=1 cell is +1.2 points nominal
|
||||
(~3 items at n=256, not significant), the rest below. Removing the
|
||||
mixed-task interference bought ~2 points over adapter_uni (9.4 -> 11.7
|
||||
at k=1) — interference was real but marginal, not masking a capability.
|
||||
The GSM8K chapter is closed: the recipe family's best member, trained
|
||||
on GSM alone in the correct regime, delivers no usable gain and the
|
||||
standard fidelity damage; combined with the scope note, the boundary
|
||||
claim (structural: supervision density + state-evolution bottleneck)
|
||||
is fully supported.
|
||||
|
||||
Reference in New Issue
Block a user