201 lines
9.2 KiB
Markdown
201 lines
9.2 KiB
Markdown
# gpuq — bucket-backed multi-node GPU job queue
|
||
|
||
A minimal scheduler for keeping N GPUs across M hosts busy when the hosts
|
||
cannot reach each other directly (rented vast.ai nodes) but all share one
|
||
S3 bucket (rclone remote `jspace`, Hetzner). Three shell scripts, no
|
||
daemon, no database, no cross-node networking.
|
||
|
||
```
|
||
scripts/gpuq_worker.sh one per GPU runs jobs from its own queue
|
||
scripts/gpuq_supervisor.sh one per node restarts workers, heartbeats
|
||
scripts/gpuq_submit.sh anywhere enqueue jobs / show status
|
||
scripts/gpuq_dispatcher.sh ONE, always-on assigns class jobs to idle GPUs
|
||
```
|
||
|
||
## Design in one paragraph
|
||
|
||
Every GPU gets a private queue directory in the bucket:
|
||
`jspace:jspace/gpuq/<worker-id>/pending/`. A worker loop polls its own
|
||
directory (20 s), pulls job files, runs them oldest-first-by-name with
|
||
`CUDA_VISIBLE_DEVICES` pinned, and moves job + full log to `done/`.
|
||
There is **no work stealing and no job claiming** — each queue has exactly
|
||
one consumer, so the S3 lack of atomic move can never double-run a job.
|
||
Load balancing happens at submit time (you pick the worker); ordering
|
||
happens in the filename (`00_`, `01_`, …). The bucket is both transport
|
||
and durable state: a dead node's pending jobs stay visible and can be
|
||
moved to another worker with one `rclone move`.
|
||
|
||
## Bucket layout
|
||
|
||
```
|
||
gpuq/
|
||
_health/<node-id>.txt heartbeat, rewritten every 60 s
|
||
<node-id>-gpu<i>/
|
||
pending/<job>.sh waiting (consumed oldest-first by name)
|
||
done/<job>.sh finished job file
|
||
done/<job>.log its full stdout+stderr, ends with "rc=N"
|
||
```
|
||
|
||
Local mirror on the node: `~/gpuq/<worker-id>/` (pending copy, logs).
|
||
|
||
## Node setup (once per node, after git clone/pull of ~/jspace)
|
||
|
||
Requirements: `rclone` configured with the `jspace` remote (see
|
||
`.s3-credentials.txt` on the Spark — do NOT commit it), tmux session
|
||
`ssh_tmux`, GPUs visible to `nvidia-smi`.
|
||
|
||
Start ONLY the supervisor — it spawns and maintains all workers itself:
|
||
|
||
```bash
|
||
tmux new-window -t ssh_tmux -n gpuq_sup \
|
||
'bash ~/jspace/scripts/gpuq_supervisor.sh node1 5' # node-id, #GPUs
|
||
```
|
||
|
||
The supervisor, every 60 s:
|
||
1. respawns any missing `gpuq_worker.sh node1-gpu<i>` as tmux window
|
||
`gpuq<i>` (nohup fallback if tmux is unreachable);
|
||
2. publishes `gpuq/_health/node1.txt`: timestamp, uptime, per-GPU
|
||
util/mem, disk on `/` and `/dev/shm`, live worker count, local
|
||
pending count.
|
||
|
||
There is deliberately no supervisor-of-supervisors: a stale heartbeat
|
||
(> ~2 min) IS the signal that the node or supervisor died — check it
|
||
from outside.
|
||
|
||
## Job contract
|
||
|
||
A job is a plain bash file. The worker provides `CUDA_VISIBLE_DEVICES`;
|
||
everything else is the job's business. Template:
|
||
|
||
```bash
|
||
#!/bin/bash
|
||
cd ~/jspace/scripts
|
||
P=~/jspace/.venv/bin/python # /venv/main/bin/python on vast images
|
||
export HF_HUB_OFFLINE=1 # model already cached
|
||
LOOP_OUT=~/jspace/results-loop-12b $P train_merge_code.py --seed 7 --alpha 0.15
|
||
rclone copy ~/jspace/results-loop-12b jspace:jspace/results-12b/ \
|
||
--include "adapter_code_s7*" --include "*.json" -q
|
||
```
|
||
|
||
Rules of thumb:
|
||
- **Jobs sync their own results** to the bucket at the end (the queue only
|
||
ships the job's log automatically, not its artifacts).
|
||
- One GPU per job; never set CUDA_VISIBLE_DEVICES yourself.
|
||
- Idempotence is on the submitter: if you resubmit a job, it reruns.
|
||
- Exit code lands as the last line of the log (`rc=0`).
|
||
|
||
## Submitting and monitoring (from any machine with the remote)
|
||
|
||
```bash
|
||
# ordered pair on one GPU (train, then eval — order = filename sort):
|
||
scripts/gpuq_submit.sh node1-gpu0 00_train_a15.sh 01_eval_a15.sh
|
||
|
||
# fan five seeds across five GPUs:
|
||
for s in 0 1 2 3 4; do
|
||
scripts/gpuq_submit.sh node1-gpu$s seed$s.sh
|
||
done
|
||
|
||
scripts/gpuq_submit.sh --status # pending + last 20 finished
|
||
rclone cat jspace:jspace/gpuq/_health/node1.txt # node health
|
||
rclone cat jspace:jspace/gpuq/node1-gpu0/done/00_train_a15.log | tail -50
|
||
```
|
||
|
||
## Dependencies
|
||
|
||
- Same-GPU sequential dependency: filename prefixes on one worker
|
||
(`00_train.sh`, `01_eval.sh`). This covers nearly all our chains.
|
||
- Cross-GPU/cross-node dependencies: not supported by design. Either
|
||
submit the dependent job after seeing the first finish in `--status`,
|
||
or make the job itself poll the bucket for its input artifact before
|
||
starting (a `until rclone lsf ...; do sleep 60; done` preamble).
|
||
|
||
## Failure modes and recovery
|
||
|
||
| failure | effect | recovery |
|
||
|---|---|---|
|
||
| job crashes (rc≠0) | worker moves on to next job | read `done/<job>.log`; resubmit fixed job |
|
||
| worker dies | its queue stalls | supervisor respawns it ≤ 60 s |
|
||
| supervisor dies | workers keep running; heartbeat goes stale | restart supervisor window |
|
||
| node dies | heartbeat stale; pending jobs preserved in bucket | `rclone move gpuq/node1-gpu0/pending gpuq/node2-gpu3/pending` |
|
||
| duplicate submit | job runs twice (queues are dumb) | submitter's responsibility |
|
||
| partial upload read | impossible — S3 uploads are atomic (objects appear only complete) | — |
|
||
|
||
Local `~/gpuq/` is scratch; the bucket is the source of truth.
|
||
|
||
## Why not X (considered alternatives)
|
||
|
||
- **Slurm/K8s**: assume stable, mutually-reachable nodes; wrong shape for
|
||
ephemeral single-tenant rentals.
|
||
- **Ray / SkyPilot / dstack**: viable (dstack has native vast.ai support
|
||
and is the thing to try for the *next* provisioning burst), but they
|
||
own provisioning; gpuq feeds nodes that already exist.
|
||
- **iroh p2p binary**: would give push dispatch, live log streaming, and
|
||
direct node↔node transfer through NAT (QUIC hole-punching) — attractive
|
||
if the fleet becomes permanent. Rejected for now: jobs run 10 min–2 h,
|
||
so poll latency is irrelevant, and durable queue state in S3 comes free
|
||
vs. a coordinator protocol we'd have to write and debug. For ad-hoc live
|
||
log streaming, n0's prebuilt `dumbpipe` works without writing code.
|
||
- **Marker-file watchers / pkill chains** (our previous approach): see
|
||
LESSONS.md — self-matching kill patterns and watcher pileups burned us
|
||
repeatedly. gpuq replaces markers with per-consumer queues and replaces
|
||
pkill with the supervisor owning worker lifecycle.
|
||
|
||
## Security notes
|
||
|
||
The bucket credentials on rented nodes are readable by the host operator:
|
||
keep the `jspace` remote scoped to this project's bucket, and rotate the
|
||
key after each rental burst (already flagged in vast-ai-notes.md).
|
||
|
||
---
|
||
|
||
## As deployed (2026-07-14) — operator cheat sheet
|
||
|
||
**Fleet:**
|
||
|
||
| node | workers | supervisor lives in |
|
||
|---|---|---|
|
||
| `node3` (8×H100, vast) | `node3-gpu0` … `node3-gpu7` | tmux `ssh_tmux:gpuq_sup` |
|
||
| `node2` (2×H100, vast) | `node2-gpu0`, `node2-gpu1` | tmux `ssh_tmux:gpuq_sup` (drain watcher armed on the Spark) |
|
||
| `spark` (GB10) | `spark-gpu0` | detached, tmux session `gpuq_spark` |
|
||
|
||
**Daily usage (from the Spark or anywhere with the `jspace` remote):**
|
||
|
||
```bash
|
||
bash scripts/gpuq_submit.sh --class h100 myjob.sh # PREFERRED: class queue;
|
||
# dispatcher assigns to the
|
||
# next idle GPU in class
|
||
bash scripts/gpuq_submit.sh node3-gpu4 myjob.sh # pin to a specific GPU
|
||
# (ordered chains only)
|
||
bash scripts/gpuq_submit.sh --status # pending + last 20 done
|
||
rclone cat jspace:jspace/gpuq/_health/node3.txt # heartbeat; stale >2 min = trouble
|
||
rclone cat jspace:jspace/gpuq/node3-gpu4/done/myjob.log | tail -30
|
||
```
|
||
|
||
**Job conventions (learned the hard way — see LESSONS.md):**
|
||
|
||
1. First line of work: `cd <repo> && git pull origin main -q` — jobs
|
||
self-update; never rely on what a node cloned at boot (repo is public,
|
||
no keys needed).
|
||
2. `HF_HUB_OFFLINE=1` ONLY if every model/dataset the job touches is already
|
||
cached on that node — it blocks `datasets` downloads too (bit us twice).
|
||
3. Jobs end with their own `rclone copy` of artifacts to the bucket; the
|
||
queue ships only the log automatically.
|
||
4. Env the job must set itself: venv activation (`/venv/main` on vast,
|
||
`~/jspace/.venv` on the Spark), `JLENS_MODEL`, `JLENS_BAND`, `LOOP_OUT`,
|
||
`HF_HOME`. The worker provides exactly one thing: `CUDA_VISIBLE_DEVICES`.
|
||
5. Ordering on one GPU = filename sort (`00_train.sh`, `01_eval.sh`).
|
||
Cross-GPU deps: submit after the prerequisite shows in `--status`, or
|
||
add an `until rclone lsf <artifact>; do sleep 60; done` preamble.
|
||
6. **Class scheduling**: `--class` drops jobs into
|
||
`gpuq/_class-pending/<class>/`; the single dispatcher (runs on the Spark,
|
||
`gpuq_dispatcher.sh`) moves each job to a concrete worker the moment one
|
||
is idle (empty pending + GPU util <15% + heartbeat fresher than 3 min).
|
||
Rosters: `gpuq/_classes/<class>.txt`, one worker id per line — edit with
|
||
`rclone`; adding/removing fleet nodes touches only these files. One
|
||
dispatcher only: it is the single writer that makes class queues
|
||
race-free. If it dies, class jobs simply wait; restart it anywhere.
|
||
7. Draining a node: stop submitting to its queues, wait for `--status` to
|
||
clear, `pkill -f gpuq_`, final `rclone copy` of its results dirs, then
|
||
destroy the instance. Pending jobs of a dead node survive in the bucket:
|
||
`rclone move jspace:jspace/gpuq/nodeX-gpuN/pending jspace:jspace/gpuq/nodeY-gpuM/pending`.
|