HumanEval transfer: trained-vs-untrained-merge paired test (p=0.80, n.s.) -- transfer gain is the merge itself, not trained content

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-14 16:30:36 +02:00
parent d1bafc2d4d
commit dcd0b0b2dc
4 changed files with 17 additions and 4 deletions
@@ -183,6 +183,9 @@ def main():
         arm_maps["HumanEval loop k=4"][0], None),
        ("HumanEval distill k=1 vs k=0, overall", he_tr0,
         arm_maps["HumanEval distill k=1"][0], None),
+        ("HumanEval trained vs UNTRAINED merge (k=2), overall",
+         per_item("eval_humaneval_untrained.json", 2),
+         per_item("eval_humaneval_trained.json", 2), None),
    ]
    rust0 = arm_maps["Rust transfer k=0"][0]
    rust4 = arm_maps["Rust transfer k=4"][0]