diff --git a/PAPER.md b/PAPER.md index e84bce2..faa340f 100644 --- a/PAPER.md +++ b/PAPER.md @@ -278,10 +278,13 @@ changes with scale, and per-task tuning remains unavoidable. ![Transfer panel](results-loop/fig_transfer.png) MBPP-trained implants applied unchanged: **HumanEval** overall 58.5%→66.5% -(loop k=4, p=0.011; hard 0→31.6%); notably the *untrained* merge already -reaches 64.6% and the transferred pause adapter 66.5% (hard 38.9%) — the -transfer is substrate-shaped (a generically useful perturbation+content -mode), not task-memorized. **Rust/MultiPL-E** (Python-trained, different +(loop k=4, p=0.011 vs base; hard 0→31.6%). The decisive control: the +*untrained* merge already reaches 64.6%, and trained-vs-untrained is **not +significant** (paired McNemar at k=2, 9 vs 7 discordant, p=0.80). What +transfers significantly is the *merge perturbation itself*, not the +MBPP-trained content — the cleanest evidence that off-distribution value is +substrate-shaped rather than task-memorized. (The transferred pause adapter +reaches 66.5%, hard 38.9%, consistent with the same reading.) **Rust/MultiPL-E** (Python-trained, different language, compile-run-verified): hard 8.0%→24.0% (p=0.125 at n=25 — directionally consistent, underpowered). **Blocksworld** MBPP-transfer: hard 0→14.3% (task-trained: 43%). Content transfers where the substrate's diff --git a/results-loop/STATS.md b/results-loop/STATS.md index 422e2cb..c926b59 100644 --- a/results-loop/STATS.md +++ b/results-loop/STATS.md @@ -27,6 +27,7 @@ - stack-train k=4 vs distill k=1, hard: A-only 10, B-only 6, n=55, p=0.4545 (n.s.) - HumanEval loop k=4 vs k=0, overall: A-only 5, B-only 18, n=164, p=0.01062 (**significant**) - HumanEval distill k=1 vs k=0, overall: A-only 7, B-only 17, n=164, p=0.06391 (n.s.) +- HumanEval trained vs UNTRAINED merge (k=2), overall: A-only 7, B-only 9, n=164, p=0.8036 (n.s.) - Rust loop k=4 vs k=0, overall: A-only 11, B-only 5, n=154, p=0.2101 (n.s.) - Rust loop k=4 vs k=0, hard: A-only 0, B-only 4, n=25, p=0.125 (n.s.) diff --git a/results-loop/stats_final.json b/results-loop/stats_final.json index 5278d67..2a7234a 100644 --- a/results-loop/stats_final.json +++ b/results-loop/stats_final.json @@ -305,6 +305,12 @@ "b_only": 17, "p": 0.06391465663909912 }, + "HumanEval trained vs UNTRAINED merge (k=2), overall": { + "n": 164, + "a_only": 7, + "b_only": 9, + "p": 0.803619384765625 + }, "Rust loop k=4 vs k=0, overall": { "n": 154, "a_only": 11, diff --git a/scripts/stats_final.py b/scripts/stats_final.py index c6ecd64..cf09c16 100644 --- a/scripts/stats_final.py +++ b/scripts/stats_final.py @@ -183,6 +183,9 @@ def main(): arm_maps["HumanEval loop k=4"][0], None), ("HumanEval distill k=1 vs k=0, overall", he_tr0, arm_maps["HumanEval distill k=1"][0], None), + ("HumanEval trained vs UNTRAINED merge (k=2), overall", + per_item("eval_humaneval_untrained.json", 2), + per_item("eval_humaneval_trained.json", 2), None), ] rust0 = arm_maps["Rust transfer k=0"][0] rust4 = arm_maps["Rust transfer k=4"][0]