nils/jspace
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

gpuq: idle-time class dispatcher (single-writer, race-free) + docs

Browse Source 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Nils
2026-07-14 12:47:23 +02:00
co-authored by Claude Fable 5
parent 9329f22a63
commit cd8e94dda5
3 changed files with 57 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/GPUQ.md
+18 -5
View File
@@ -6,9 +6,10 @@ S3 bucket (rclone remote `jspace`, Hetzner). Three shell scripts, no
daemon, no database, no cross-node networking.
```
scripts/gpuq_worker.sh one per GPU runs jobs from its own queue
scripts/gpuq_supervisor.sh one per node restarts workers, heartbeats
scripts/gpuq_submit.sh anywhere enqueue jobs / show status
scripts/gpuq_worker.sh one per GPU runs jobs from its own queue
scripts/gpuq_supervisor.sh one per node restarts workers, heartbeats
scripts/gpuq_submit.sh anywhere enqueue jobs / show status
scripts/gpuq_dispatcher.sh ONE, always-on assigns class jobs to idle GPUs
```
## Design in one paragraph
@@ -160,7 +161,11 @@ key after each rental burst (already flagged in vast-ai-notes.md).
**Daily usage (from the Spark or anywhere with the `jspace` remote):**
```bash
bash scripts/gpuq_submit.sh node3-gpu4 myjob.sh # submit to a specific GPU
bash scripts/gpuq_submit.sh --class h100 myjob.sh # PREFERRED: class queue;
# dispatcher assigns to the
# next idle GPU in class
bash scripts/gpuq_submit.sh node3-gpu4 myjob.sh # pin to a specific GPU
# (ordered chains only)
bash scripts/gpuq_submit.sh --status # pending + last 20 done
rclone cat jspace:jspace/gpuq/_health/node3.txt # heartbeat; stale >2 min = trouble
rclone cat jspace:jspace/gpuq/node3-gpu4/done/myjob.log | tail -30
@@ -181,7 +186,15 @@ rclone cat jspace:jspace/gpuq/node3-gpu4/done/myjob.log | tail -30
5. Ordering on one GPU = filename sort (`00_train.sh`, `01_eval.sh`).
Cross-GPU deps: submit after the prerequisite shows in `--status`, or
add an `until rclone lsf <artifact>; do sleep 60; done` preamble.
6. Draining a node: stop submitting to its queues, wait for `--status` to
6. **Class scheduling**: `--class` drops jobs into
`gpuq/_class-pending/<class>/`; the single dispatcher (runs on the Spark,
`gpuq_dispatcher.sh`) moves each job to a concrete worker the moment one
is idle (empty pending + GPU util <15% + heartbeat fresher than 3 min).
Rosters: `gpuq/_classes/<class>.txt`, one worker id per line — edit with
`rclone`; adding/removing fleet nodes touches only these files. One
dispatcher only: it is the single writer that makes class queues
race-free. If it dies, class jobs simply wait; restart it anywhere.
7. Draining a node: stop submitting to its queues, wait for `--status` to
clear, `pkill -f gpuq_`, final `rclone copy` of its results dirs, then
destroy the instance. Pending jobs of a dead node survive in the bucket:
`rclone move jspace:jspace/gpuq/nodeX-gpuN/pending jspace:jspace/gpuq/nodeY-gpuM/pending`.