# gpuq — bucket-backed multi-node GPU job queue A minimal scheduler for keeping N GPUs across M hosts busy when the hosts cannot reach each other directly (rented vast.ai nodes) but all share one S3 bucket (rclone remote `jspace`, Hetzner). Three shell scripts, no daemon, no database, no cross-node networking. ``` scripts/gpuq_worker.sh one per GPU runs jobs from its own queue scripts/gpuq_supervisor.sh one per node restarts workers, heartbeats scripts/gpuq_submit.sh anywhere enqueue jobs / show status scripts/gpuq_dispatcher.sh ONE, always-on assigns class jobs to idle GPUs ``` ## Design in one paragraph Every GPU gets a private queue directory in the bucket: `jspace:jspace/gpuq//pending/`. A worker loop polls its own directory (20 s), pulls job files, runs them oldest-first-by-name with `CUDA_VISIBLE_DEVICES` pinned, and moves job + full log to `done/`. There is **no work stealing and no job claiming** — each queue has exactly one consumer, so the S3 lack of atomic move can never double-run a job. Load balancing happens at submit time (you pick the worker); ordering happens in the filename (`00_`, `01_`, …). The bucket is both transport and durable state: a dead node's pending jobs stay visible and can be moved to another worker with one `rclone move`. ## Data contract (selective sync) Jobs declare what they need and what they produce as comments; the worker does the rest — no node needs everything, and results survive node death: ``` # gpuq-in: results-loop/mbpp_data.json results-loop/adapter_code.pt # gpuq-out: results-loop/eval_code_gate*.json results/jbar_31b.pt ``` Paths are repo-relative, mirrored at `jspace:jspace/`. Inputs are pulled before the job starts. Outputs are pushed every `$GPUQ_SYNC_EVERY` seconds (default 120) while the job runs and once at exit — a dying node loses at most one interval. The in-flight log is also mirrored to `gpuq//running/` each cycle. Globs allowed in basenames; repeat lines for more paths; jobs with no declarations sync nothing (backward compatible). Manual counterpart from any machine: ``` scripts/gpuq_sync.sh pull results-loop "eval_code_gate*.json" scripts/gpuq_sync.sh push results-loop "adapter_code.pt" ``` Nodes need the rclone remote configured for any of this; keep a scoped (write-limited) key for rented nodes. Without credentials the worker still runs jobs — sync lines just no-op. ## Bucket layout ``` gpuq/ _health/.txt heartbeat, rewritten every 60 s -gpu/ pending/.sh waiting (consumed oldest-first by name) done/.sh finished job file done/.log its full stdout+stderr, ends with "rc=N" ``` Local mirror on the node: `~/gpuq//` (pending copy, logs). ## Node setup (once per node, after git clone/pull of ~/jspace) Requirements: `rclone` configured with the `jspace` remote (see `.s3-credentials.txt` on the Spark — do NOT commit it), tmux session `ssh_tmux`, GPUs visible to `nvidia-smi`. Start ONLY the supervisor — it spawns and maintains all workers itself: ```bash tmux new-window -t ssh_tmux -n gpuq_sup \ 'bash ~/jspace/scripts/gpuq_supervisor.sh node1 5' # node-id, #GPUs ``` The supervisor, every 60 s: 1. respawns any missing `gpuq_worker.sh node1-gpu` as tmux window `gpuq` (nohup fallback if tmux is unreachable); 2. publishes `gpuq/_health/node1.txt`: timestamp, uptime, per-GPU util/mem, disk on `/` and `/dev/shm`, live worker count, local pending count. There is deliberately no supervisor-of-supervisors: a stale heartbeat (> ~2 min) IS the signal that the node or supervisor died — check it from outside. ## Job contract A job is a plain bash file. The worker provides `CUDA_VISIBLE_DEVICES`; everything else is the job's business. Template: ```bash #!/bin/bash cd ~/jspace/scripts P=~/jspace/.venv/bin/python # /venv/main/bin/python on vast images export HF_HUB_OFFLINE=1 # model already cached LOOP_OUT=~/jspace/results-loop-12b $P train_merge_code.py --seed 7 --alpha 0.15 rclone copy ~/jspace/results-loop-12b jspace:jspace/results-12b/ \ --include "adapter_code_s7*" --include "*.json" -q ``` Rules of thumb: - **Jobs sync their own results** to the bucket at the end (the queue only ships the job's log automatically, not its artifacts). - One GPU per job; never set CUDA_VISIBLE_DEVICES yourself. - Idempotence is on the submitter: if you resubmit a job, it reruns. - Exit code lands as the last line of the log (`rc=0`). ## Submitting and monitoring (from any machine with the remote) ```bash # ordered pair on one GPU (train, then eval — order = filename sort): scripts/gpuq_submit.sh node1-gpu0 00_train_a15.sh 01_eval_a15.sh # fan five seeds across five GPUs: for s in 0 1 2 3 4; do scripts/gpuq_submit.sh node1-gpu$s seed$s.sh done scripts/gpuq_submit.sh --status # pending + last 20 finished rclone cat jspace:jspace/gpuq/_health/node1.txt # node health rclone cat jspace:jspace/gpuq/node1-gpu0/done/00_train_a15.log | tail -50 ``` ## Dependencies - Same-GPU sequential dependency: filename prefixes on one worker (`00_train.sh`, `01_eval.sh`). This covers nearly all our chains. - Cross-GPU/cross-node dependencies: not supported by design. Either submit the dependent job after seeing the first finish in `--status`, or make the job itself poll the bucket for its input artifact before starting (a `until rclone lsf ...; do sleep 60; done` preamble). ## Failure modes and recovery | failure | effect | recovery | |---|---|---| | job crashes (rc≠0) | worker moves on to next job | read `done/.log`; resubmit fixed job | | worker dies | its queue stalls | supervisor respawns it ≤ 60 s | | supervisor dies | workers keep running; heartbeat goes stale | restart supervisor window | | node dies | heartbeat stale; pending jobs preserved in bucket | `rclone move gpuq/node1-gpu0/pending gpuq/node2-gpu3/pending` | | duplicate submit | job runs twice (queues are dumb) | submitter's responsibility | | partial upload read | impossible — S3 uploads are atomic (objects appear only complete) | — | Local `~/gpuq/` is scratch; the bucket is the source of truth. ## Why not X (considered alternatives) - **Slurm/K8s**: assume stable, mutually-reachable nodes; wrong shape for ephemeral single-tenant rentals. - **Ray / SkyPilot / dstack**: viable (dstack has native vast.ai support and is the thing to try for the *next* provisioning burst), but they own provisioning; gpuq feeds nodes that already exist. - **iroh p2p binary**: would give push dispatch, live log streaming, and direct node↔node transfer through NAT (QUIC hole-punching) — attractive if the fleet becomes permanent. Rejected for now: jobs run 10 min–2 h, so poll latency is irrelevant, and durable queue state in S3 comes free vs. a coordinator protocol we'd have to write and debug. For ad-hoc live log streaming, n0's prebuilt `dumbpipe` works without writing code. - **Marker-file watchers / pkill chains** (our previous approach): see LESSONS.md — self-matching kill patterns and watcher pileups burned us repeatedly. gpuq replaces markers with per-consumer queues and replaces pkill with the supervisor owning worker lifecycle. ## Security notes The bucket credentials on rented nodes are readable by the host operator: keep the `jspace` remote scoped to this project's bucket, and rotate the key after each rental burst (already flagged in vast-ai-notes.md). --- ## As deployed (2026-07-14) — operator cheat sheet **Fleet:** | node | workers | supervisor lives in | |---|---|---| | `node3` (8×H100, vast) | `node3-gpu0` … `node3-gpu7` | tmux `ssh_tmux:gpuq_sup` | | `node2` (2×H100, vast) | `node2-gpu0`, `node2-gpu1` | tmux `ssh_tmux:gpuq_sup` (drain watcher armed on the Spark) | | `spark` (GB10) | `spark-gpu0` | detached, tmux session `gpuq_spark` | **Daily usage (from the Spark or anywhere with the `jspace` remote):** ```bash bash scripts/gpuq_submit.sh --class h100 myjob.sh # PREFERRED: class queue; # dispatcher assigns to the # next idle GPU in class bash scripts/gpuq_submit.sh node3-gpu4 myjob.sh # pin to a specific GPU # (ordered chains only) bash scripts/gpuq_submit.sh --status # pending + last 20 done rclone cat jspace:jspace/gpuq/_health/node3.txt # heartbeat; stale >2 min = trouble rclone cat jspace:jspace/gpuq/node3-gpu4/done/myjob.log | tail -30 ``` **Job conventions (learned the hard way — see LESSONS.md):** 1. First line of work: `cd && git pull origin main -q` — jobs self-update; never rely on what a node cloned at boot (repo is public, no keys needed). 2. `HF_HUB_OFFLINE=1` ONLY if every model/dataset the job touches is already cached on that node — it blocks `datasets` downloads too (bit us twice). 3. Jobs end with their own `rclone copy` of artifacts to the bucket; the queue ships only the log automatically. 4. Env the job must set itself: venv activation (`/venv/main` on vast, `~/jspace/.venv` on the Spark), `JLENS_MODEL`, `JLENS_BAND`, `LOOP_OUT`, `HF_HOME`. The worker provides exactly one thing: `CUDA_VISIBLE_DEVICES`. 5. Ordering on one GPU = filename sort (`00_train.sh`, `01_eval.sh`). Cross-GPU deps: submit after the prerequisite shows in `--status`, or add an `until rclone lsf ; do sleep 60; done` preamble. 6. **Class scheduling**: `--class` drops jobs into `gpuq/_class-pending//`; the single dispatcher (runs on the Spark, `gpuq_dispatcher.sh`) moves each job to a concrete worker the moment one is idle (empty pending + GPU util <15% + heartbeat fresher than 3 min). Rosters: `gpuq/_classes/.txt`, one worker id per line — edit with `rclone`; adding/removing fleet nodes touches only these files. One dispatcher only: it is the single writer that makes class queues race-free. If it dies, class jobs simply wait; restart it anywhere. 7. Draining a node: stop submitting to its queues, wait for `--status` to clear, `pkill -f gpuq_`, final `rclone copy` of its results dirs, then destroy the instance. Pending jobs of a dead node survive in the bucket: `rclone move jspace:jspace/gpuq/nodeX-gpuN/pending jspace:jspace/gpuq/nodeY-gpuM/pending`.