n=500 controls: paired net-effect significant (loop vs untrained merge 17-4, p=0.007); ladder and net accounting finalized

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Nils
2026-07-14 20:08:57 +02:00
co-authored by Claude Fable 5
parent 1d4cffd6ad
commit 0fd93cb328
8 changed files with 10160 additions and 19 deletions
+17 -12
View File
@@ -22,10 +22,11 @@ pooled across MBPP, HumanEval, and Rust/MultiPL-E, the plan-dependent bucket
moves from 4.2% to 35.6% (McNemar p≈1.5e-10) — and **placement is decisive,
not a convenience**: the gain appears only when the loop enters at the
lens-identified boundary (L14), collapsing at L13 and below and structurally
nulling above. Roughly a third of the gross effect is generic perturbation
(the untrained merge alone reaches ~18%; the bucket conditions on k=0
failure, so regression-to-mean contributes to any intervention); the
loop-specific net effect is ~+20 points over that floor.
nulling above. A substantial part of the gross effect is generic
perturbation — the untrained merge alone reaches 20.0% on the same bucket
(it conditions on k=0 failure, so regression-to-mean contributes to any
intervention) — but the loop-specific net effect over that floor is real:
paired on items, loop-vs-untrained-merge is 174 discordant (p=0.007).
A complete attribution program then deflates the mechanism's mystique: the
loop's content is *amortizable* (plan-distillation into the same adapter,
@@ -70,7 +71,9 @@ not laws.)*
three benchmarks, 4.2%→35.6%, p≈1.5e-10. Overall accuracy is
statistically unchanged on MBPP (p=0.34) and improved on HumanEval
transfer (58.5%→66.5%, p=0.011). Net of the untrained-merge floor
(~18%), the loop-specific effect is ~+20 points.
(20.0% at n=55), the loop-specific effect is +17.5 points (seed mean)
and **survives the paired test** (loop vs untrained merge on hard,
p=0.007; distill vs untrained, p=0.0075).
3. **A deflationary mechanism finding.** The trained loop converges to a
fixed point by k≈34 and behaves as *amortized plan content*, not
@@ -191,8 +194,8 @@ MBPP hard bucket (plan-dependent, n=55 unless noted):
| arm | hard pass@1 | overall |
|---|---|---|
| base (k=0, bit-exact) | 5.5% | 51.8% |
| untrained loop (α-merge only, n=28) | 17.9% | ~52% |
| trained FF, no recurrence (n=28) | 17.9% | ~52% |
| untrained loop (α-merge only, k=4) | 20.0% | 50.2% |
| trained FF, no recurrence (k=1) | 27.3% | 53.6% |
| pause-16 registers (width) | 36.4% | 55.2% |
| **trained loop k=4** (seed mean, 5 seeds) | **37.5±5.5** (best 43.6) | 53.6% |
| rung-2: + entrance-faded band LoRA (n=28) | 42.9/46.4 (2 seeds) | 51.2/52.4 |
@@ -209,11 +212,13 @@ ordering is not.
**Net accounting.** The attribution-critical comparison is trained-loop vs
*untrained merge*, not vs base: gross 5.5→37.5 (seed mean), of which the
untrained perturbation floor is ~18 points — the loop-specific net effect
is ~+20 points. The untrained-loop and trained-FF control rows above are
from the 250-item era (hard n=28); full-bucket (n=55) reruns of both
controls, enabling the paired loop-vs-untrained test, are running and will
replace these rows. [CONTROLS-N55 PENDING]
untrained perturbation floor is 20.0 points — the loop-specific net is
+17.5 (seed mean) / +23.6 (best seed), and the paired item-level test is
decisive (loop-only 17, untrained-only 4, p=0.007; distill likewise
p=0.0075). The trained-FF control (27.3%) sits between floor and loop,
not significantly above the floor (p=0.48): weights alone buy little
without either recurrence or plan supervision. All controls now n=500 /
hard n=55, same harness.
### 3.3 The decisive tests: nothing stacks