item 17 scored: GSM-only current recipe fails as predicted; GSM chapter closed
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -345,3 +345,17 @@ levels — only seed means are.
|
|||||||
(noise-s0, h2048) variants have no mechanism by which task change
|
(noise-s0, h2048) variants have no mechanism by which task change
|
||||||
could invert their MBPP verdict. Only the recipe family's best member
|
could invert their MBPP verdict. Only the recipe family's best member
|
||||||
(this item) is informative on GSM8K.
|
(this item) is informative on GSM8K.
|
||||||
|
|
||||||
|
--- Outcome, item 17 (GSM-only, current recipe, scored 2026-07-15):
|
||||||
|
predictions (a) and (b) CONFIRMED, (c) nominally exceeded but not
|
||||||
|
meaningfully. hard 8.7/5.5/4.7% at k=1/2/4 (below the 10% bar; nowhere
|
||||||
|
near the 15% reopen threshold); easy 93.1 -> 48-52% at k>0; overall
|
||||||
|
11.7/10.9/10.2 vs k0's 10.5 — the k=1 cell is +1.2 points nominal
|
||||||
|
(~3 items at n=256, not significant), the rest below. Removing the
|
||||||
|
mixed-task interference bought ~2 points over adapter_uni (9.4 -> 11.7
|
||||||
|
at k=1) — interference was real but marginal, not masking a capability.
|
||||||
|
The GSM8K chapter is closed: the recipe family's best member, trained
|
||||||
|
on GSM alone in the correct regime, delivers no usable gain and the
|
||||||
|
standard fidelity damage; combined with the scope note, the boundary
|
||||||
|
claim (structural: supervision density + state-evolution bottleneck)
|
||||||
|
is fully supported.
|
||||||
|
|||||||
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user