nils/jspace
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

GPUQ: as-deployed cheat sheet + job conventions

Browse Source 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Nils
2026-07-14 12:40:39 +02:00
co-authored by Claude Fable 5
parent d2d372c909
commit c718c636a5
1 changed files with 41 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/GPUQ.md
+41
View File
@@ -144,3 +144,44 @@ Local `~/gpuq/` is scratch; the bucket is the source of truth.
The bucket credentials on rented nodes are readable by the host operator: The bucket credentials on rented nodes are readable by the host operator:
keep the `jspace` remote scoped to this project's bucket, and rotate the keep the `jspace` remote scoped to this project's bucket, and rotate the
key after each rental burst (already flagged in vast-ai-notes.md). key after each rental burst (already flagged in vast-ai-notes.md).
---
## As deployed (2026-07-14) — operator cheat sheet
**Fleet:**
| node | workers | supervisor lives in |
|---|---|---|
| `node3` (8×H100, vast) | `node3-gpu0` … `node3-gpu7` | tmux `ssh_tmux:gpuq_sup` |
| `node2` (2×H100, vast) | `node2-gpu0`, `node2-gpu1` | tmux `ssh_tmux:gpuq_sup` (drain watcher armed on the Spark) |
| `spark` (GB10) | `spark-gpu0` | detached, tmux session `gpuq_spark` |
**Daily usage (from the Spark or anywhere with the `jspace` remote):**
```bash
bash scripts/gpuq_submit.sh node3-gpu4 myjob.sh # submit to a specific GPU
bash scripts/gpuq_submit.sh --status # pending + last 20 done
rclone cat jspace:jspace/gpuq/_health/node3.txt # heartbeat; stale >2 min = trouble
rclone cat jspace:jspace/gpuq/node3-gpu4/done/myjob.log | tail -30
```
**Job conventions (learned the hard way — see LESSONS.md):**
1. First line of work: `cd <repo> && git pull origin main -q` — jobs
self-update; never rely on what a node cloned at boot (repo is public,
no keys needed).
2. `HF_HUB_OFFLINE=1` ONLY if every model/dataset the job touches is already
cached on that node — it blocks `datasets` downloads too (bit us twice).
3. Jobs end with their own `rclone copy` of artifacts to the bucket; the
queue ships only the log automatically.
4. Env the job must set itself: venv activation (`/venv/main` on vast,
`~/jspace/.venv` on the Spark), `JLENS_MODEL`, `JLENS_BAND`, `LOOP_OUT`,
`HF_HOME`. The worker provides exactly one thing: `CUDA_VISIBLE_DEVICES`.
5. Ordering on one GPU = filename sort (`00_train.sh`, `01_eval.sh`).
Cross-GPU deps: submit after the prerequisite shows in `--status`, or
add an `until rclone lsf <artifact>; do sleep 60; done` preamble.
6. Draining a node: stop submitting to its queues, wait for `--status` to
clear, `pkill -f gpuq_`, final `rclone copy` of its results dirs, then
destroy the instance. Pending jobs of a dead node survive in the bucket:
`rclone move jspace:jspace/gpuq/nodeX-gpuN/pending jspace:jspace/gpuq/nodeY-gpuM/pending`.