LESSONS: thermal hard-freeze pattern and diagnosis by exclusion
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
+9
-1
@@ -62,7 +62,15 @@ reproduction: [`RESULTS.md`](RESULTS.md). Everything on `google/gemma-4-E2B-it`
|
||||
6. **Background jobs must be `setsid`'d** or the harness/session restart
|
||||
kills them mid-run. And `pkill -f <pattern>` will match your own launcher
|
||||
shell if the pattern appears in its command line.
|
||||
7. **Zero-init adapter output layer ⇒ zero grads upstream at step 0** — on
|
||||
7. **Sustained training in a closed cabinet = thermal hard-freezes.** Four
|
||||
crashes in one day (journal stops mid-line, no OOM, no shutdown trace,
|
||||
37GB free at one death) on a DGX Spark that was stable all week under
|
||||
light load. Pattern: only under hours of continuous GPU load; fixed by
|
||||
opening the cabinet. Diagnose by exclusion: earlyoom quiet + journal
|
||||
truncation + load correlation = thermal, not software. And do NOT
|
||||
auto-restart training via @reboot cron on a thermally-suspect box — it
|
||||
risks a crash loop with no human circuit breaker.
|
||||
8. **Zero-init adapter output layer ⇒ zero grads upstream at step 0** — on
|
||||
`mlp[0]` this is expected (LoRA-B-style), not a bug; check the output
|
||||
layer's grad instead.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user