diff --git a/scripts/GPUQ.md b/scripts/GPUQ.md index beecb75..b8928e2 100644 --- a/scripts/GPUQ.md +++ b/scripts/GPUQ.md @@ -144,3 +144,44 @@ Local `~/gpuq/` is scratch; the bucket is the source of truth. The bucket credentials on rented nodes are readable by the host operator: keep the `jspace` remote scoped to this project's bucket, and rotate the key after each rental burst (already flagged in vast-ai-notes.md). + +--- + +## As deployed (2026-07-14) — operator cheat sheet + +**Fleet:** + +| node | workers | supervisor lives in | +|---|---|---| +| `node3` (8×H100, vast) | `node3-gpu0` … `node3-gpu7` | tmux `ssh_tmux:gpuq_sup` | +| `node2` (2×H100, vast) | `node2-gpu0`, `node2-gpu1` | tmux `ssh_tmux:gpuq_sup` (drain watcher armed on the Spark) | +| `spark` (GB10) | `spark-gpu0` | detached, tmux session `gpuq_spark` | + +**Daily usage (from the Spark or anywhere with the `jspace` remote):** + +```bash +bash scripts/gpuq_submit.sh node3-gpu4 myjob.sh # submit to a specific GPU +bash scripts/gpuq_submit.sh --status # pending + last 20 done +rclone cat jspace:jspace/gpuq/_health/node3.txt # heartbeat; stale >2 min = trouble +rclone cat jspace:jspace/gpuq/node3-gpu4/done/myjob.log | tail -30 +``` + +**Job conventions (learned the hard way — see LESSONS.md):** + +1. First line of work: `cd && git pull origin main -q` — jobs + self-update; never rely on what a node cloned at boot (repo is public, + no keys needed). +2. `HF_HUB_OFFLINE=1` ONLY if every model/dataset the job touches is already + cached on that node — it blocks `datasets` downloads too (bit us twice). +3. Jobs end with their own `rclone copy` of artifacts to the bucket; the + queue ships only the log automatically. +4. Env the job must set itself: venv activation (`/venv/main` on vast, + `~/jspace/.venv` on the Spark), `JLENS_MODEL`, `JLENS_BAND`, `LOOP_OUT`, + `HF_HOME`. The worker provides exactly one thing: `CUDA_VISIBLE_DEVICES`. +5. Ordering on one GPU = filename sort (`00_train.sh`, `01_eval.sh`). + Cross-GPU deps: submit after the prerequisite shows in `--status`, or + add an `until rclone lsf ; do sleep 60; done` preamble. +6. Draining a node: stop submitting to its queues, wait for `--status` to + clear, `pkill -f gpuq_`, final `rclone copy` of its results dirs, then + destroy the instance. Pending jobs of a dead node survive in the bucket: + `rclone move jspace:jspace/gpuq/nodeX-gpuN/pending jspace:jspace/gpuq/nodeY-gpuM/pending`.