n=500 controls: paired net-effect significant (loop vs untrained merge 17-4, p=0.007); ladder and net accounting finalized
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -22,10 +22,11 @@ pooled across MBPP, HumanEval, and Rust/MultiPL-E, the plan-dependent bucket
|
||||
moves from 4.2% to 35.6% (McNemar p≈1.5e-10) — and **placement is decisive,
|
||||
not a convenience**: the gain appears only when the loop enters at the
|
||||
lens-identified boundary (L14), collapsing at L13 and below and structurally
|
||||
nulling above. Roughly a third of the gross effect is generic perturbation
|
||||
(the untrained merge alone reaches ~18%; the bucket conditions on k=0
|
||||
failure, so regression-to-mean contributes to any intervention); the
|
||||
loop-specific net effect is ~+20 points over that floor.
|
||||
nulling above. A substantial part of the gross effect is generic
|
||||
perturbation — the untrained merge alone reaches 20.0% on the same bucket
|
||||
(it conditions on k=0 failure, so regression-to-mean contributes to any
|
||||
intervention) — but the loop-specific net effect over that floor is real:
|
||||
paired on items, loop-vs-untrained-merge is 17–4 discordant (p=0.007).
|
||||
|
||||
A complete attribution program then deflates the mechanism's mystique: the
|
||||
loop's content is *amortizable* (plan-distillation into the same adapter,
|
||||
@@ -70,7 +71,9 @@ not laws.)*
|
||||
three benchmarks, 4.2%→35.6%, p≈1.5e-10. Overall accuracy is
|
||||
statistically unchanged on MBPP (p=0.34) and improved on HumanEval
|
||||
transfer (58.5%→66.5%, p=0.011). Net of the untrained-merge floor
|
||||
(~18%), the loop-specific effect is ~+20 points.
|
||||
(20.0% at n=55), the loop-specific effect is +17.5 points (seed mean)
|
||||
and **survives the paired test** (loop vs untrained merge on hard,
|
||||
p=0.007; distill vs untrained, p=0.0075).
|
||||
|
||||
3. **A deflationary mechanism finding.** The trained loop converges to a
|
||||
fixed point by k≈3–4 and behaves as *amortized plan content*, not
|
||||
@@ -191,8 +194,8 @@ MBPP hard bucket (plan-dependent, n=55 unless noted):
|
||||
| arm | hard pass@1 | overall |
|
||||
|---|---|---|
|
||||
| base (k=0, bit-exact) | 5.5% | 51.8% |
|
||||
| untrained loop (α-merge only, n=28) | 17.9% | ~52% |
|
||||
| trained FF, no recurrence (n=28) | 17.9% | ~52% |
|
||||
| untrained loop (α-merge only, k=4) | 20.0% | 50.2% |
|
||||
| trained FF, no recurrence (k=1) | 27.3% | 53.6% |
|
||||
| pause-16 registers (width) | 36.4% | 55.2% |
|
||||
| **trained loop k=4** (seed mean, 5 seeds) | **37.5±5.5** (best 43.6) | 53.6% |
|
||||
| rung-2: + entrance-faded band LoRA (n=28) | 42.9/46.4 (2 seeds) | 51.2/52.4 |
|
||||
@@ -209,11 +212,13 @@ ordering is not.
|
||||
|
||||
**Net accounting.** The attribution-critical comparison is trained-loop vs
|
||||
*untrained merge*, not vs base: gross 5.5→37.5 (seed mean), of which the
|
||||
untrained perturbation floor is ~18 points — the loop-specific net effect
|
||||
is ~+20 points. The untrained-loop and trained-FF control rows above are
|
||||
from the 250-item era (hard n=28); full-bucket (n=55) reruns of both
|
||||
controls, enabling the paired loop-vs-untrained test, are running and will
|
||||
replace these rows. [CONTROLS-N55 PENDING]
|
||||
untrained perturbation floor is 20.0 points — the loop-specific net is
|
||||
+17.5 (seed mean) / +23.6 (best seed), and the paired item-level test is
|
||||
decisive (loop-only 17, untrained-only 4, p=0.007; distill likewise
|
||||
p=0.0075). The trained-FF control (27.3%) sits between floor and loop,
|
||||
not significantly above the floor (p=0.48): weights alone buy little
|
||||
without either recurrence or plan supervision. All controls now n=500 /
|
||||
hard n=55, same harness.
|
||||
|
||||
### 3.3 The decisive tests: nothing stacks
|
||||
|
||||
|
||||
Reference in New Issue
Block a user