item 20 scored; item 21 (GSM carry-CoT + control) pre-registered and queued

2026-07-16 02:42:46 +02:00
parent 4b9e168838
commit 1618c206ae
9 changed files with 7431 additions and 0 deletions
@@ -454,3 +454,43 @@ started. Next per plan: threshold sweep (cheap) before any E2.
    overall >= 55% — if so, gate-quality headroom is large and further
    gate work is justified; if oracle < 53%, gating this merge is nearly
    saturated and the program pivots to E2 or closes.
+
+--- Outcome, item 20 (scored 2026-07-16 ~02:35): (b) CONFIRMED — clean
+monotone threshold curve (hard 7->50%, easy 96.7->87.7%, E[k] 0.43->2.63
+across theta .3->.99). (a) FAILED — no theta reaches easy>=93 AND
+hard>=32; at matched easy the E0 frozen probe dominates the entire
+learned-head curve: the BCE-trained 3K head is strictly worse than the
+class-balanced logistic probe it was meant to replace. (c) CONFIRMED,
+emphatically: ORACLE gate = 59.6 overall / easy 100% / hard 64.3% at
+E[k]=0.24. Key insight: hard items are DEPTH-DIVERSE — 18/28 solvable at
+some k in {0,1,2,4} but no single k solves more than 13; a third of the
+hard bucket lives in per-item depth selection. Program continues per
+rule; binding constraint quantified: gate quality is worth ~9.6 overall
+points (50.0 deployed vs 59.6 oracle). Also noted: the LUT re-run of the
+canonical merge shows small systematic drift vs the Jul-13 eval (k4 hard
+46.4 identical, k1/k2 hard 3 items lower) — the LUT (per-item, single
+harness run) is now the canonical reference. Next candidates, in cost
+order: (i) deploy E0's probe AS the gate against the LUT (free,
+offline); (ii) stronger classifier (multi-position features, more data,
+calibrated threshold); (iii) oracle-gap error analysis on the hard items
+no fixed k solves but some k does.
+
+21. **E2 stage A: dense short-CoT supervision through the carry
+    whiteboard, GSM8K (pre-registered 2026-07-16 ~02:55, before running;
+    PLAN_SELFPACED E2 / the hybrid from the internalization discussion).**
+    Prep: harvest TERSE verified CoTs ("at most 3 short steps", answer-
+    verified, STaR filter) for GSM train. Arms: (A) carry regime
+    (k=2 prefill, pauses easy p=2 / hard p=6) trained with CE on
+    scratchpad+answer (~30-60 dense tokens — the ingredient every latent
+    GSM arm lacked); (B) CONTROL: identical supervision, feedforward
+    adapter, no recurrence. Eval: GSM test 256, grid 0:0 (base), 2:2,
+    2:6; e400 checkpoints. Predictions: (a) arm A beats every previous
+    GSM arm's overall (>12.1%) — dense supervision is the binding fix;
+    (b) the A-vs-B delta isolates the whiteboard: if A > B by >=3 points
+    overall, recurrence adds value beyond visible-scratchpad training;
+    if A ~= B, the scratchpad text alone carries it (deflation, GSM
+    edition); (c) easy-bucket damage smaller than answer-only carry's
+    (83->45%) because training and deployment output formats now match.
+    Honest note: arm outputs are VISIBLE tokens (~40) — this is the
+    budget-CoT-with-loop hybrid, a scope change from latent planning,
+    run at Nils's explicit direction ("do gsm8k and such").
@@ -0,0 +1,93 @@
+{
+ "curve": [
+  {
+   "theta": 0.3,
+   "overall": 0.488,
+   "by_label": {
+    "easy": 0.9672131147540983,
+    "hard": 0.07142857142857142,
+    "drop": 0.02
+   },
+   "ek": 0.432
+  },
+  {
+   "theta": 0.5,
+   "overall": 0.5,
+   "by_label": {
+    "easy": 0.9590163934426229,
+    "hard": 0.17857142857142858,
+    "drop": 0.03
+   },
+   "ek": 0.74
+  },
+  {
+   "theta": 0.7,
+   "overall": 0.496,
+   "by_label": {
+    "easy": 0.9426229508196722,
+    "hard": 0.17857142857142858,
+    "drop": 0.04
+   },
+   "ek": 1.136
+  },
+  {
+   "theta": 0.8,
+   "overall": 0.508,
+   "by_label": {
+    "easy": 0.9344262295081968,
+    "hard": 0.2857142857142857,
+    "drop": 0.05
+   },
+   "ek": 1.388
+  },
+  {
+   "theta": 0.9,
+   "overall": 0.496,
+   "by_label": {
+    "easy": 0.8852459016393442,
+    "hard": 0.39285714285714285,
+    "drop": 0.05
+   },
+   "ek": 1.76
+  },
+  {
+   "theta": 0.95,
+   "overall": 0.492,
+   "by_label": {
+    "easy": 0.8770491803278688,
+    "hard": 0.42857142857142855,
+    "drop": 0.04
+   },
+   "ek": 2.084
+  },
+  {
+   "theta": 0.98,
+   "overall": 0.504,
+   "by_label": {
+    "easy": 0.8770491803278688,
+    "hard": 0.4642857142857143,
+    "drop": 0.06
+   },
+   "ek": 2.412
+  },
+  {
+   "theta": 0.99,
+   "overall": 0.508,
+   "by_label": {
+    "easy": 0.8770491803278688,
+    "hard": 0.5,
+    "drop": 0.06
+   },
+   "ek": 2.628
+  }
+ ],
+ "oracle": {
+  "overall": 0.596,
+  "by_label": {
+   "easy": 1.0,
+   "hard": 0.6428571428571429,
+   "drop": 0.09
+  },
+  "ek": 0.236
+ }
+}