gpuq data contract: declarative per-job in/out sync (pull inputs, push outputs every 120s + at exit), gpuq_sync.sh helper

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Nils
2026-07-16 01:17:33 +02:00
co-authored by Claude Fable 5
parent 684e793eef
commit 550d5cbe56
3 changed files with 93 additions and 6 deletions
+27
View File
@@ -25,6 +25,33 @@ happens in the filename (`00_`, `01_`, …). The bucket is both transport
and durable state: a dead node's pending jobs stay visible and can be
moved to another worker with one `rclone move`.
## Data contract (selective sync)
Jobs declare what they need and what they produce as comments; the worker
does the rest — no node needs everything, and results survive node death:
```
# gpuq-in: results-loop/mbpp_data.json results-loop/adapter_code.pt
# gpuq-out: results-loop/eval_code_gate*.json results/jbar_31b.pt
```
Paths are repo-relative, mirrored at `jspace:jspace/<same path>`.
Inputs are pulled before the job starts. Outputs are pushed every
`$GPUQ_SYNC_EVERY` seconds (default 120) while the job runs and once at
exit — a dying node loses at most one interval. The in-flight log is
also mirrored to `gpuq/<worker>/running/` each cycle. Globs allowed in
basenames; repeat lines for more paths; jobs with no declarations sync
nothing (backward compatible). Manual counterpart from any machine:
```
scripts/gpuq_sync.sh pull results-loop "eval_code_gate*.json"
scripts/gpuq_sync.sh push results-loop "adapter_code.pt"
```
Nodes need the rclone remote configured for any of this; keep a scoped
(write-limited) key for rented nodes. Without credentials the worker
still runs jobs — sync lines just no-op.
## Bucket layout
```