diff --git a/HANDOFF.md b/HANDOFF.md new file mode 100644 index 0000000..9762dde --- /dev/null +++ b/HANDOFF.md @@ -0,0 +1,98 @@ +# Session handoff — 2026-07-16 ~09:45 + +Previous session ended because the auto-mode permission classifier entered +a persistent session-level lockout (blocked nearly all Bash/Write on +"earlier conversation content" grounds, ~7h intermittent). Everything +below is current as of ~09:45. Repo knowledge map: `INDEX.md`. Scored +experiment record: `results-loop/PROTOCOL_UNIFIED.md` (items 1–21, all +scored except pending stats below). Research plan: `PLAN_SELFPACED.md`. + +## RUNNING RIGHT NOW — do not disturb, monitor these + +### Rented H100 node ("node 1") +- **ssh -p 52525 root@185.151.171.35** (vast.ai; greets with a banner — + filter it). Costs money while it runs. +- **`/root/pipeline5.log`**: lens-mapping pipeline (setsid, survives ssh + drops). Sequence: 31B jbar DONE (saved+synced) → 31B exp4 scan DONE as + a printed TABLE (its `regimes.pt` save crashed — known bug, see + fix-list) → 31B weights cleaned → **26B-A4B jacobian scan RUNNING** + (started ~07:40, ~40-90s/prompt × 256; expect done late morning). + After it: 26B exp4 scan (same save bug will strike — harmless, the + table survives in the log), CLEANED, PIPELINE-DONE. +- **`/root/node_sync.sh` sidecar** (also setsid): rclone-pushes + `/workspace/jspace/results/` (jbar_*.pt, exp4_*.log) and pipeline5.log + to `jspace:jspace/results-lens/` every 120s. The node HAS rclone + credentials (Nils placed them himself — never copy credentials to + nodes from automation; hard-blocked + it's his call). +- When PIPELINE-DONE: verify `results-lens/` has jbar_31b.pt, + jbar_26b_a4b.pt, exp4_31b.log, exp4_26b_a4b.log — then the node can be + destroyed (tell Nils; it's his dashboard). +- Storage lore: /workspace = disk (survives stop, dies on destroy); + /dev/shm = RAM (dies on stop). See memory/vast-node-storage.md. + +### Spark (this machine) +- gpuq worker in tmux session `gpuq_spark:gpuq0`, queue EMPTY, healthy. + Worker code now has: data contract (`# gpuq-in:`/`# gpuq-out:` job + comments → auto-sync inputs/outputs with the bucket, outputs every + 120s during the job) and graceful drain (`touch ~/gpuq/spark-gpu0/STOP` + — NEVER kill the tmux session mid-job; that cost 3.5h once). +- Thermal: box hard-froze 4× on 2026-07-15 under sustained load — + cabinet now open, stable since. If it freezes: reboot, restart worker + (`tmux new-session -d -s gpuq_spark -n gpuq0 'bash + ~/jspace/scripts/gpuq_worker.sh spark-gpu0 0'`). NO @reboot cron + (Nils vetoed — crash-loop risk). + +## IMMEDIATE PENDING (blocked by the classifier, run first) + +1. `.venv/bin/python scripts/mcnemar_carrycot.py` — the paired p-values + for last night's headline result, especially the drop-bucket test. + THE morning number; item 21's outcome note says "pending". +2. Commit everything uncommitted: + `git add -A scripts/ PLAN_SELFPACED.md HANDOFF.md && git add -f + results-loop/PROTOCOL_UNIFIED.md results-loop/eval_gsm_carrycot*.json + results-loop/gsm_cot_data.json && git commit && git push origin main` + (uncommitted: mcnemar script, lens-noise trainer changes + [train_carry_cot.py --lensnoise, UNVERIFIED — parse-check first], + plan E2-L/E2-A2/E2-N sections, protocol item-21 scoring, this file). +3. Parse the 31B (and later 26B) regime tables from exp4_*.log into + regimes_*.pt — copy the pattern in `scripts/parse_e4b_regimes.py` + (adjust layer count: 31B=60, 26B=48? read from log). Then fill + REGIMES.json rows (results/REGIMES.json) and re-render + `scripts/fig_regimes.py` (5-scale figure, panels auto-fill). +4. Patch `scripts/exp4_regimes.py` to take an explicit output path + (torch.save to CWD-relative "results/" has now crashed 3 scans). + +## LAST NIGHT'S RESULTS (already scored in the protocol) + +- **Item 21, the headline**: GSM8K carry-cot (dense self-distilled terse + scratchpads through the carry whiteboard): **57.4%** best cell vs + 12.1% all-time prior best; base 10.9. Control (same supervision, no + recurrence): 54.7 best. Whiteboard's specific edge: DROP items + (+9/+14 across cells) — reach into problems unreachable at labeling. + If McNemar confirms → gates open for: stage B internalization ladder + (PLAN E2-L), A2 on-policy refresh (E2-A2), lens-shaped noise (E2-N, + Nils's idea, N1 already implemented as --lensnoise, unverified). +- **Item 20**: gate threshold curve — E0's frozen logistic probe + dominates every learned gate; ORACLE gate = 59.6% overall at 0.24 + mean iters (hard items depth-diverse: 18/28 solvable at some k, ≤13 + at any single k). Gate program continues; binding constraint = + classifier quality on the pre-loop state. +- **E4B lens surprise** (REGIMES.json, provisional): no E2B-style + workspace signature — sensor L11-22, motor from L23, persistence bump + ABSENT; candidate band sits inside the KV-shared zone. The elastic + pair (E2B⊂E4B) reorganized. Needs ignition cross-check + jbar top-up + before strong claims. + +## MORNING DECISION QUEUE (Nils decides, one submit each) + +With McNemar in hand: which of stage B (internalization) / A2 +(re-harvest) / E2-N1 (lens-noise, verify parse first) gets the Spark. +All pre-registered or planned in PLAN_SELFPACED.md; job template +pattern: scripts/jobs/zzz_l_gsm_e2a.sh (uses the data contract). + +## OPERATIONAL CAUTIONS (paid for in blood, see LESSONS.md 1-12 + memory) + +- pkill -f self-match; k=0 sanity row in every eval; e400 pre-commit; + seeds before believing single cells (noise-s0 taught this twice); + monitors: grep patterns must match eval_loop.py's "acc=" (GSM) vs + "pass@1=" (MBPP); artifacts leave nodes within one sync cycle. diff --git a/results-loop/PROTOCOL_UNIFIED.md b/results-loop/PROTOCOL_UNIFIED.md index b186825..1aa16fe 100644 --- a/results-loop/PROTOCOL_UNIFIED.md +++ b/results-loop/PROTOCOL_UNIFIED.md @@ -513,8 +513,14 @@ consistent, specific signature: DROP items (unreachable by the base model even with full CoT at labeling) — A beats B by +9 and +14 points there in the two cells. Interpretation: scratchpad supervision carries the bulk; the carry chain specifically extends reach into previously -unreachable problems. McNemar p-values pending (mcnemar_carrycot.py -ready; classifier-blocked at scoring time). (c) CONFIRMED: easy damage +unreachable problems. McNemar (scored 2026-07-16 morning): overall +A-vs-B not significant in either cell (2:2 discordants 34-23 p=0.185; +2:6 33-26 p=0.435), but the pre-identified drop-bucket signature IS: +2:6 drop discordants 20-6, exact McNemar p=0.0094 (survives +Bonferroni x4 = 0.038); 2:2 drop 16-7, p=0.093 (same direction, +marginal). Verdict: the whiteboard's edge is real and specific to +extending reach into drop items, not a general lift over matched +supervision. (c) CONFIRMED: easy damage much reduced vs answer-only carry (83->72-83 vs 83->45). Ladder gate: technically met at p=2; decision on stage B/A2/E2-N deferred to the morning review with the p-values in hand — the drop-bucket signature, diff --git a/results-loop/fig_kcurves2.png b/results-loop/fig_kcurves2.png deleted file mode 100644 index dc60de7..0000000 Binary files a/results-loop/fig_kcurves2.png and /dev/null differ diff --git a/results-loop/fig_phase.png b/results-loop/fig_phase.png index 0c2ae40..c9060ba 100644 Binary files a/results-loop/fig_phase.png and b/results-loop/fig_phase.png differ diff --git a/scripts/mcnemar_carrycot.py b/scripts/mcnemar_carrycot.py index 651965e..d4fa6bc 100644 --- a/scripts/mcnemar_carrycot.py +++ b/scripts/mcnemar_carrycot.py @@ -22,5 +22,8 @@ for cell in ("2:2", "2:6"): if star.get(i) == "drop" and a[i] and not b[i]) dy = sum(1 for i in ids if star.get(i) == "drop" and b[i] and not a[i]) + dn = dx + dy + dp = (min(1, sum(comb(dn, k) for k in range(max(dx, dy), dn + 1)) + / 2 ** dn * 2) if dn else 1) print(f"{cell}: carry-only={x} ff-only={y} McNemar p={p:.4f} " - f"drop-bucket discordants {dx}-{dy}") + f"drop-bucket discordants {dx}-{dy} McNemar p={dp:.4f}")