nils/jspace
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

LESSONS: thermal hard-freeze pattern and diagnosis by exclusion

Browse Source 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Nils
2026-07-15 18:30:01 +02:00
co-authored by Claude Fable 5
parent b17d7faea0
commit b6d54c2726
1 changed files with 9 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
LESSONS.md
+9 -1
View File
@@ -62,7 +62,15 @@ reproduction: [`RESULTS.md`](RESULTS.md). Everything on `google/gemma-4-E2B-it`
6. **Background jobs must be `setsid`'d** or the harness/session restart 6. **Background jobs must be `setsid`'d** or the harness/session restart
kills them mid-run. And `pkill -f <pattern>` will match your own launcher kills them mid-run. And `pkill -f <pattern>` will match your own launcher
shell if the pattern appears in its command line. shell if the pattern appears in its command line.
7. **Zero-init adapter output layer ⇒ zero grads upstream at step 0** — on 7. **Sustained training in a closed cabinet = thermal hard-freezes.** Four
crashes in one day (journal stops mid-line, no OOM, no shutdown trace,
37GB free at one death) on a DGX Spark that was stable all week under
light load. Pattern: only under hours of continuous GPU load; fixed by
opening the cabinet. Diagnose by exclusion: earlyoom quiet + journal
truncation + load correlation = thermal, not software. And do NOT
auto-restart training via @reboot cron on a thermally-suspect box — it
risks a crash loop with no human circuit breaker.
8. **Zero-init adapter output layer ⇒ zero grads upstream at step 0** — on
`mlp[0]` this is expected (LoRA-B-style), not a bug; check the output `mlp[0]` this is expected (LoRA-B-style), not a bug; check the output
layer's grad instead. layer's grad instead.