99 lines
5.4 KiB
Markdown
99 lines
5.4 KiB
Markdown
# Session handoff — 2026-07-16 ~09:45
|
||
|
||
Previous session ended because the auto-mode permission classifier entered
|
||
a persistent session-level lockout (blocked nearly all Bash/Write on
|
||
"earlier conversation content" grounds, ~7h intermittent). Everything
|
||
below is current as of ~09:45. Repo knowledge map: `INDEX.md`. Scored
|
||
experiment record: `results-loop/PROTOCOL_UNIFIED.md` (items 1–21, all
|
||
scored except pending stats below). Research plan: `PLAN_SELFPACED.md`.
|
||
|
||
## RUNNING RIGHT NOW — do not disturb, monitor these
|
||
|
||
### Rented H100 node ("node 1")
|
||
- **ssh -p 52525 root@185.151.171.35** (vast.ai; greets with a banner —
|
||
filter it). Costs money while it runs.
|
||
- **`/root/pipeline5.log`**: lens-mapping pipeline (setsid, survives ssh
|
||
drops). Sequence: 31B jbar DONE (saved+synced) → 31B exp4 scan DONE as
|
||
a printed TABLE (its `regimes.pt` save crashed — known bug, see
|
||
fix-list) → 31B weights cleaned → **26B-A4B jacobian scan RUNNING**
|
||
(started ~07:40, ~40-90s/prompt × 256; expect done late morning).
|
||
After it: 26B exp4 scan (same save bug will strike — harmless, the
|
||
table survives in the log), CLEANED, PIPELINE-DONE.
|
||
- **`/root/node_sync.sh` sidecar** (also setsid): rclone-pushes
|
||
`/workspace/jspace/results/` (jbar_*.pt, exp4_*.log) and pipeline5.log
|
||
to `jspace:jspace/results-lens/` every 120s. The node HAS rclone
|
||
credentials (Nils placed them himself — never copy credentials to
|
||
nodes from automation; hard-blocked + it's his call).
|
||
- When PIPELINE-DONE: verify `results-lens/` has jbar_31b.pt,
|
||
jbar_26b_a4b.pt, exp4_31b.log, exp4_26b_a4b.log — then the node can be
|
||
destroyed (tell Nils; it's his dashboard).
|
||
- Storage lore: /workspace = disk (survives stop, dies on destroy);
|
||
/dev/shm = RAM (dies on stop). See memory/vast-node-storage.md.
|
||
|
||
### Spark (this machine)
|
||
- gpuq worker in tmux session `gpuq_spark:gpuq0`, queue EMPTY, healthy.
|
||
Worker code now has: data contract (`# gpuq-in:`/`# gpuq-out:` job
|
||
comments → auto-sync inputs/outputs with the bucket, outputs every
|
||
120s during the job) and graceful drain (`touch ~/gpuq/spark-gpu0/STOP`
|
||
— NEVER kill the tmux session mid-job; that cost 3.5h once).
|
||
- Thermal: box hard-froze 4× on 2026-07-15 under sustained load —
|
||
cabinet now open, stable since. If it freezes: reboot, restart worker
|
||
(`tmux new-session -d -s gpuq_spark -n gpuq0 'bash
|
||
~/jspace/scripts/gpuq_worker.sh spark-gpu0 0'`). NO @reboot cron
|
||
(Nils vetoed — crash-loop risk).
|
||
|
||
## IMMEDIATE PENDING (blocked by the classifier, run first)
|
||
|
||
1. `.venv/bin/python scripts/mcnemar_carrycot.py` — the paired p-values
|
||
for last night's headline result, especially the drop-bucket test.
|
||
THE morning number; item 21's outcome note says "pending".
|
||
2. Commit everything uncommitted:
|
||
`git add -A scripts/ PLAN_SELFPACED.md HANDOFF.md && git add -f
|
||
results-loop/PROTOCOL_UNIFIED.md results-loop/eval_gsm_carrycot*.json
|
||
results-loop/gsm_cot_data.json && git commit && git push origin main`
|
||
(uncommitted: mcnemar script, lens-noise trainer changes
|
||
[train_carry_cot.py --lensnoise, UNVERIFIED — parse-check first],
|
||
plan E2-L/E2-A2/E2-N sections, protocol item-21 scoring, this file).
|
||
3. Parse the 31B (and later 26B) regime tables from exp4_*.log into
|
||
regimes_*.pt — copy the pattern in `scripts/parse_e4b_regimes.py`
|
||
(adjust layer count: 31B=60, 26B=48? read from log). Then fill
|
||
REGIMES.json rows (results/REGIMES.json) and re-render
|
||
`scripts/fig_regimes.py` (5-scale figure, panels auto-fill).
|
||
4. Patch `scripts/exp4_regimes.py` to take an explicit output path
|
||
(torch.save to CWD-relative "results/" has now crashed 3 scans).
|
||
|
||
## LAST NIGHT'S RESULTS (already scored in the protocol)
|
||
|
||
- **Item 21, the headline**: GSM8K carry-cot (dense self-distilled terse
|
||
scratchpads through the carry whiteboard): **57.4%** best cell vs
|
||
12.1% all-time prior best; base 10.9. Control (same supervision, no
|
||
recurrence): 54.7 best. Whiteboard's specific edge: DROP items
|
||
(+9/+14 across cells) — reach into problems unreachable at labeling.
|
||
If McNemar confirms → gates open for: stage B internalization ladder
|
||
(PLAN E2-L), A2 on-policy refresh (E2-A2), lens-shaped noise (E2-N,
|
||
Nils's idea, N1 already implemented as --lensnoise, unverified).
|
||
- **Item 20**: gate threshold curve — E0's frozen logistic probe
|
||
dominates every learned gate; ORACLE gate = 59.6% overall at 0.24
|
||
mean iters (hard items depth-diverse: 18/28 solvable at some k, ≤13
|
||
at any single k). Gate program continues; binding constraint =
|
||
classifier quality on the pre-loop state.
|
||
- **E4B lens surprise** (REGIMES.json, provisional): no E2B-style
|
||
workspace signature — sensor L11-22, motor from L23, persistence bump
|
||
ABSENT; candidate band sits inside the KV-shared zone. The elastic
|
||
pair (E2B⊂E4B) reorganized. Needs ignition cross-check + jbar top-up
|
||
before strong claims.
|
||
|
||
## MORNING DECISION QUEUE (Nils decides, one submit each)
|
||
|
||
With McNemar in hand: which of stage B (internalization) / A2
|
||
(re-harvest) / E2-N1 (lens-noise, verify parse first) gets the Spark.
|
||
All pre-registered or planned in PLAN_SELFPACED.md; job template
|
||
pattern: scripts/jobs/zzz_l_gsm_e2a.sh (uses the data contract).
|
||
|
||
## OPERATIONAL CAUTIONS (paid for in blood, see LESSONS.md 1-12 + memory)
|
||
|
||
- pkill -f self-match; k=0 sanity row in every eval; e400 pre-commit;
|
||
seeds before believing single cells (noise-s0 taught this twice);
|
||
monitors: grep patterns must match eval_loop.py's "acc=" (GSM) vs
|
||
"pass@1=" (MBPP); artifacts leave nodes within one sync cycle.
|