jspace/vast-ai-notes.md

Vast.ai node runbook — lessons from the 2×H200 session (2026-07-14)

What actually works, and every trap we hit getting there. Cost of learning: ~80 idle GPU-minutes. Next setup should take <15 minutes.

The recipe that works

# 1) image: Vast "PyTorch" template (torch in /venv/main, NOT system python)
source /venv/main/bin/activate
uv pip install -q "transformers==5.13.*" datasets accelerate hf_transfer

# 2) work in RAM — tmpfs beats the tiny container disk (check: df -h /dev/shm)
export HF_HOME=/dev/shm/hf
mkdir -p /dev/shm/jspace

# 3) code: rsync the LIVE working tree from the dev box — never a stale tarball
#    (from dev box:)  rsync -a -e "ssh -p PORT" jspace/scripts jspace/jlens root@IP:/dev/shm/jspace/

# 4) data/artifacts: rclone from the bucket (config: scp ~/.config/rclone/rclone.conf over)
rclone copy jspace:jspace/results-loop /dev/shm/jspace/results-loop/

# 5) models: hf_transfer or bust (see pitfall #1)
export HF_TOKEN=... HF_HUB_ENABLE_HF_TRANSFER=1
hf download google/gemma-4-12B-it            # ~2.5 GB/s vs stalls without

# 6) AFTER downloads complete: force offline so nothing ever waits on network
export HF_HUB_OFFLINE=1 HF_DATASETS_OFFLINE=1
#    (datasets caches rsync over from dev box: ~/.cache/huggingface/datasets/<name>)

# 7) EVERYTHING long-running goes in tmux windows — that is the deal
tmux new-window -t ssh_tmux -n myjob 'bash /dev/shm/run_myjob.sh'
#    pipe stages through `tee` so panes show live output AND files exist for pollers

# 8) results survive teardown via a sync loop (tmpfs is volatile):
while true; do rclone copy /dev/shm/jspace/results-X jspace:jspace/results-X/; sleep 300; done

Pitfalls, in the order they bit us

1. Single-stream HF downloads stall dead on some hosts (twice: at 9.6 GB and 18 GB, zero error, zero timeout). pip install hf_transfer + HF_HUB_ENABLE_HF_TRANSFER=1 fixed it instantly (~2.5 GB/s). Always.
2. Never kill a downloader mid-finalization. Killing hf download while it "verifies" left a snapshot missing tokenizer.json → prompts tokenized to <unk>, generations decoded to None/empty, every labeling pass silently produced 0.000 accuracy. Diagnosis that found it: md5sum/file diff of the snapshot dir vs a known-good cache. A later hf download re-run did NOT restore the missing file (it trusted the snapshot); explicitly downloading the single file did: hf download <repo> tokenizer.json.
3. Stale .locks deadlock everything after any killed HF process. rm -rf $HF_HOME/hub/.locks before relaunching. Symptom: "Still waiting to acquire lock… (elapsed: NNs)" forever.
4. Two processes downloading the same model = lock contention. Download once, THEN start parallel jobs.
5. pkill -f <pattern> kills your own command if the pattern appears in its command line (exit code 144, half your script never runs). Use bracket-escaping: pkill -f "[h]f download". Bit us three times, twice locally, once remotely.
6. Stale code on the node. The bucket tarball was a day old; the node ran pre-patch code (wrong adapter width, wrong band, missing env handling) and produced invalid results that LOOKED like model problems. Rsync the live tree; verify with a hash or a version marker if paranoid.
7. nohup + ssh one-shots are fragile; interrupted ssh calls orphan or duplicate work, and you can't see what's happening. tmux windows in the host's ssh_tmux session: survive disconnects, visible to the human (Ctrl-b <n>), killable as a unit. Caveat: Ctrl-C in a pane kills the whole window's process group, tail included.
8. .bashrc on Vast auto-attaches tmux — sourcing it in non-interactive ssh fails ("duplicate session: ssh_tmux"). Set env inline instead.
9. Model revisions/templates differ across a family. gemma-4-12B-it uses a channel-based chat template (<|channel>thought) and ends turns with <turn|>, not E2B's <end_of_turn>. Generation-stop lists must be built per-tokenizer (filter unk!), or numeric-answer extraction reads trailing garbage. Probe ONE generation end-to-end (convert_ids_to_tokens on both prompt and output) before batch-labeling anything on a new model.
10. Probe-gate your launches. The pattern that finally worked: a tmux "fix" window that repairs → runs a single-generation probe → only on probe-pass spawns the real jobs. (And make the pass-condition robust — ours failed on a too-strict grep despite a correct generation.)
11. HF_HUB_OFFLINE=1 blocks dataset downloads too. Ship the ~/.cache/huggingface/datasets/<name> dirs from the dev box (tiny), or download datasets before going offline.

Node facts worth remembering

• Vast = unprivileged Docker container on a shared host: GPUs exclusive, CPU/RAM/network shared; the HOST OPERATOR can read everything on the box — scope and rotate any credential that touches it (HF token, bucket keys).
• vast-capabilities prints the live manifest (ports, services, creds presence). Read /etc/vast-agents-guide.md first.
• /dev/shm was 181 GB on a 1.4 TB-RAM box (container-capped) — still ample: weights + caches + outputs all in RAM; instance restart loses it, hence the rclone sync loop.
• 2×H200 NVL ran 12B labeling at ~98%/81% util with both branches parallel — roughly the Spark's E2B speed on a 5× bigger model, per GPU.