n=500 controls: paired net-effect significant (loop vs untrained merge 17-4, p=0.007); ladder and net accounting finalized

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

PAPER.md

@@ -22,10 +22,11 @@ pooled across MBPP, HumanEval, and Rust/MultiPL-E, the plan-dependent bucket
 moves from 4.2% to 35.6% (McNemar p≈1.5e-10) — and **placement is decisive,
 not a convenience**: the gain appears only when the loop enters at the
 lens-identified boundary (L14), collapsing at L13 and below and structurally
-nulling above. Roughly a third of the gross effect is generic perturbation
-(the untrained merge alone reaches ~18%; the bucket conditions on k=0
-failure, so regression-to-mean contributes to any intervention); the
-loop-specific net effect is ~+20 points over that floor.
+nulling above. A substantial part of the gross effect is generic
+perturbation — the untrained merge alone reaches 20.0% on the same bucket
+(it conditions on k=0 failure, so regression-to-mean contributes to any
+intervention) — but the loop-specific net effect over that floor is real:
+paired on items, loop-vs-untrained-merge is 17–4 discordant (p=0.007).
 
 A complete attribution program then deflates the mechanism's mystique: the
 loop's content is *amortizable* (plan-distillation into the same adapter,
@@ -70,7 +71,9 @@ not laws.)*
    three benchmarks, 4.2%→35.6%, p≈1.5e-10. Overall accuracy is
    statistically unchanged on MBPP (p=0.34) and improved on HumanEval
    transfer (58.5%→66.5%, p=0.011). Net of the untrained-merge floor
-   (~18%), the loop-specific effect is ~+20 points.
+   (20.0% at n=55), the loop-specific effect is +17.5 points (seed mean)
+   and **survives the paired test** (loop vs untrained merge on hard,
+   p=0.007; distill vs untrained, p=0.0075).
 
 3. **A deflationary mechanism finding.** The trained loop converges to a
    fixed point by k≈3–4 and behaves as *amortized plan content*, not
@@ -191,8 +194,8 @@ MBPP hard bucket (plan-dependent, n=55 unless noted):
 | arm | hard pass@1 | overall |
 |---|---|---|
 | base (k=0, bit-exact) | 5.5% | 51.8% |
-| untrained loop (α-merge only, n=28) | 17.9% | ~52% |
-| trained FF, no recurrence (n=28) | 17.9% | ~52% |
+| untrained loop (α-merge only, k=4) | 20.0% | 50.2% |
+| trained FF, no recurrence (k=1) | 27.3% | 53.6% |
 | pause-16 registers (width) | 36.4% | 55.2% |
 | **trained loop k=4** (seed mean, 5 seeds) | **37.5±5.5** (best 43.6) | 53.6% |
 | rung-2: + entrance-faded band LoRA (n=28) | 42.9/46.4 (2 seeds) | 51.2/52.4 |
@@ -209,11 +212,13 @@ ordering is not.
 
 **Net accounting.** The attribution-critical comparison is trained-loop vs
 *untrained merge*, not vs base: gross 5.5→37.5 (seed mean), of which the
-untrained perturbation floor is ~18 points — the loop-specific net effect
-is ~+20 points. The untrained-loop and trained-FF control rows above are
-from the 250-item era (hard n=28); full-bucket (n=55) reruns of both
-controls, enabling the paired loop-vs-untrained test, are running and will
-replace these rows. [CONTROLS-N55 PENDING]
+untrained perturbation floor is 20.0 points — the loop-specific net is
++17.5 (seed mean) / +23.6 (best seed), and the paired item-level test is
+decisive (loop-only 17, untrained-only 4, p=0.007; distill likewise
+p=0.0075). The trained-FF control (27.3%) sits between floor and loop,
+not significantly above the floor (p=0.48): weights alone buy little
+without either recurrence or plan supervision. All controls now n=500 /
+hard n=55, same harness.
 
 ### 3.3 The decisive tests: nothing stacks
 

results-loop/STATS.md

@@ -2,6 +2,8 @@
 
 ## Headline numbers (Wilson 95% CIs)
 
+- **MBPP untrained merge k=4 (n=500 rerun)**: overall 0.502 [0.458, 0.546] (n=500); hard 0.200 [0.116, 0.324] (n=55)
+- **MBPP trained FF k=1 (n=500 rerun)**: overall 0.536 [0.492, 0.579] (n=500); hard 0.273 [0.173, 0.402] (n=55)
 - **MBPP loop s0 k=0 (base)**: overall 0.518 [0.474, 0.561] (n=500); hard 0.055 [0.019, 0.149] (n=55)
 - **MBPP loop s0 k=2**: overall 0.530 [0.486, 0.573] (n=500); hard 0.309 [0.203, 0.440] (n=55)
 - **MBPP loop s0 k=4**: overall 0.536 [0.492, 0.579] (n=500); hard 0.436 [0.314, 0.567] (n=55)
@@ -22,6 +24,10 @@
 
 - loop k=4 vs k=0, overall: A-only 30, B-only 39, n=500, p=0.3356 (n.s.)
 - loop k=4 vs k=0, hard: A-only 1, B-only 22, n=55, p=5.722e-06 (**significant**)
+- loop k=4 vs UNTRAINED merge k=4, hard (net effect): A-only 4, B-only 17, n=55, p=0.007197 (**significant**)
+- loop k=4 vs UNTRAINED merge k=4, overall: A-only 29, B-only 46, n=500, p=0.06395 (n.s.)
+- trained FF vs UNTRAINED merge, hard: A-only 7, B-only 11, n=55, p=0.4807 (n.s.)
+- distill k=1 vs UNTRAINED merge k=4, hard: A-only 3, B-only 15, n=55, p=0.007538 (**significant**)
 - distill k=1 vs loop k=4, overall: A-only 33, B-only 37, n=500, p=0.7202 (n.s.)
 - distill k=1 vs loop k=4, hard: A-only 10, B-only 9, n=55, p=1 (n.s.)
 - stack-train k=4 vs distill k=1, hard: A-only 10, B-only 6, n=55, p=0.4545 (n.s.)

results-loop/stats_final.json

@@ -1,4 +1,40 @@
 {
+ "MBPP untrained merge k=4 (n=500 rerun)": {
+  "overall": {
+   "acc": 0.502,
+   "n": 500,
+   "ci": [
+    0.458,
+    0.546
+   ]
+  },
+  "hard": {
+   "acc": 0.2,
+   "n": 55,
+   "ci": [
+    0.116,
+    0.324
+   ]
+  }
+ },
+ "MBPP trained FF k=1 (n=500 rerun)": {
+  "overall": {
+   "acc": 0.536,
+   "n": 500,
+   "ci": [
+    0.492,
+    0.579
+   ]
+  },
+  "hard": {
+   "acc": 0.2727272727272727,
+   "n": 55,
+   "ci": [
+    0.173,
+    0.402
+   ]
+  }
+ },
  "MBPP loop s0 k=0 (base)": {
   "overall": {
    "acc": 0.518,
@@ -275,6 +311,30 @@
    "b_only": 22,
    "p": 5.7220458984375e-06
   },
+  "loop k=4 vs UNTRAINED merge k=4, hard (net effect)": {
+   "n": 55,
+   "a_only": 4,
+   "b_only": 17,
+   "p": 0.007197380065917969
+  },
+  "loop k=4 vs UNTRAINED merge k=4, overall": {
+   "n": 500,
+   "a_only": 29,
+   "b_only": 46,
+   "p": 0.06394991646706696
+  },
+  "trained FF vs UNTRAINED merge, hard": {
+   "n": 55,
+   "a_only": 7,
+   "b_only": 11,
+   "p": 0.480682373046875
+  },
+  "distill k=1 vs UNTRAINED merge k=4, hard": {
+   "n": 55,
+   "a_only": 3,
+   "b_only": 15,
+   "p": 0.007537841796875
+  },
   "distill k=1 vs loop k=4, overall": {
    "n": 500,
    "a_only": 33,

scripts/figures_final.py

@@ -100,13 +100,10 @@ def fig_ladder():
 
     _, h0 = ev(OUT / "eval_code_code_s0_full.json", 0)
     add("base (k=0, exact)", h0, NH, GRAY)
-    # untrained E2B control (250-item era, n_hard=28)
-    d = json.load(open(OUT / "eval_code_untrained.json"))
-    hu = max(v["by_label"]["hard"] for k, v in d["ks"].items() if k != "0")
-    add("untrained loop (best k)", hu, 28, GRAY)
-    d = json.load(open(OUT / "eval_code_ff.json"))
-    add("trained FF (no recurrence)", d["ks"]["1"]["by_label"]["hard"], 28,
-        ORANGE)
+    _, hu = ev(OUT / "eval_code_untrained_full.json", 4)
+    add("untrained loop (perturbation floor)", hu, NH, GRAY)
+    _, hf = ev(OUT / "eval_code_code_ff_full.json", 1)
+    add("trained FF (no recurrence)", hf, NH, ORANGE)
     _, hp = ev(OUT / "eval_code_pause16.json", 1)
     add("pause-16 registers (width)", hp, NH, ORANGE)
     _, hl = ev(OUT / "eval_code_code_s0_full.json", 4)

scripts/stats_final.py

@@ -85,6 +85,10 @@ def main():
     lines += ["## Headline numbers (Wilson 95% CIs)", ""]
     ARMS = [
         # (label, file, k, hard-subset)
+        ("MBPP untrained merge k=4 (n=500 rerun)",
+         "eval_code_untrained_full.json", 4, mbpp_hard),
+        ("MBPP trained FF k=1 (n=500 rerun)",
+         "eval_code_code_ff_full.json", 1, mbpp_hard),
         ("MBPP loop s0 k=0 (base)", "eval_code_code_s0_full.json", 0, mbpp_hard),
         ("MBPP loop s0 k=2", "eval_code_code_s0_full.json", 2, mbpp_hard),
         ("MBPP loop s0 k=4", "eval_code_code_s0_full.json", 4, mbpp_hard),
@@ -173,9 +177,16 @@ def main():
     dist1, _ = arm_maps["MBPP distill s1 k=1 (FF)"]
     stack4, _ = arm_maps["MBPP stack-train k=4"]
 
+    untr4, _ = arm_maps["MBPP untrained merge k=4 (n=500 rerun)"]
+    ff1, _ = arm_maps["MBPP trained FF k=1 (n=500 rerun)"]
     TESTS = [
         ("loop k=4 vs k=0, overall", loop0, loop4, None),
         ("loop k=4 vs k=0, hard", loop0, loop4, mbpp_hard),
+        ("loop k=4 vs UNTRAINED merge k=4, hard (net effect)",
+         untr4, loop4, mbpp_hard),
+        ("loop k=4 vs UNTRAINED merge k=4, overall", untr4, loop4, None),
+        ("trained FF vs UNTRAINED merge, hard", untr4, ff1, mbpp_hard),
+        ("distill k=1 vs UNTRAINED merge k=4, hard", untr4, dist1, mbpp_hard),
         ("distill k=1 vs loop k=4, overall", loop4, dist1, None),
         ("distill k=1 vs loop k=4, hard", loop4, dist1, mbpp_hard),
         ("stack-train k=4 vs distill k=1, hard", dist1, stack4, mbpp_hard),