HumanEval transfer: trained-vs-untrained-merge paired test (p=0.80, n.s.) -- transfer gain is the merge itself, not trained content

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

PAPER.md

@@ -278,10 +278,13 @@ changes with scale, and per-task tuning remains unavoidable.
 ![Transfer panel](results-loop/fig_transfer.png)
 
 MBPP-trained implants applied unchanged: **HumanEval** overall 58.5%→66.5%
-(loop k=4, p=0.011; hard 0→31.6%); notably the *untrained* merge already
-reaches 64.6% and the transferred pause adapter 66.5% (hard 38.9%) — the
-transfer is substrate-shaped (a generically useful perturbation+content
-mode), not task-memorized. **Rust/MultiPL-E** (Python-trained, different
+(loop k=4, p=0.011 vs base; hard 0→31.6%). The decisive control: the
+*untrained* merge already reaches 64.6%, and trained-vs-untrained is **not
+significant** (paired McNemar at k=2, 9 vs 7 discordant, p=0.80). What
+transfers significantly is the *merge perturbation itself*, not the
+MBPP-trained content — the cleanest evidence that off-distribution value is
+substrate-shaped rather than task-memorized. (The transferred pause adapter
+reaches 66.5%, hard 38.9%, consistent with the same reading.) **Rust/MultiPL-E** (Python-trained, different
 language, compile-run-verified): hard 8.0%→24.0% (p=0.125 at n=25 —
 directionally consistent, underpowered). **Blocksworld** MBPP-transfer:
 hard 0→14.3% (task-trained: 43%). Content transfers where the substrate's

results-loop/STATS.md

@@ -27,6 +27,7 @@
 - stack-train k=4 vs distill k=1, hard: A-only 10, B-only 6, n=55, p=0.4545 (n.s.)
 - HumanEval loop k=4 vs k=0, overall: A-only 5, B-only 18, n=164, p=0.01062 (**significant**)
 - HumanEval distill k=1 vs k=0, overall: A-only 7, B-only 17, n=164, p=0.06391 (n.s.)
+- HumanEval trained vs UNTRAINED merge (k=2), overall: A-only 7, B-only 9, n=164, p=0.8036 (n.s.)
 - Rust loop k=4 vs k=0, overall: A-only 11, B-only 5, n=154, p=0.2101 (n.s.)
 - Rust loop k=4 vs k=0, hard: A-only 0, B-only 4, n=25, p=0.125 (n.s.)
 

results-loop/stats_final.json

@@ -305,6 +305,12 @@
    "b_only": 17,
    "p": 0.06391465663909912
   },
+  "HumanEval trained vs UNTRAINED merge (k=2), overall": {
+   "n": 164,
+   "a_only": 7,
+   "b_only": 9,
+   "p": 0.803619384765625
+  },
   "Rust loop k=4 vs k=0, overall": {
    "n": 154,
    "a_only": 11,

scripts/stats_final.py

@@ -183,6 +183,9 @@ def main():
          arm_maps["HumanEval loop k=4"][0], None),
         ("HumanEval distill k=1 vs k=0, overall", he_tr0,
          arm_maps["HumanEval distill k=1"][0], None),
+        ("HumanEval trained vs UNTRAINED merge (k=2), overall",
+         per_item("eval_humaneval_untrained.json", 2),
+         per_item("eval_humaneval_trained.json", 2), None),
     ]
     rust0 = arm_maps["Rust transfer k=0"][0]
     rust4 = arm_maps["Rust transfer k=4"][0]