From b6d54c27266cca95e0bf4cd8510d62a634c8ff1c Mon Sep 17 00:00:00 2001 From: Nils Date: Wed, 15 Jul 2026 18:30:01 +0200 Subject: [PATCH] LESSONS: thermal hard-freeze pattern and diagnosis by exclusion Co-Authored-By: Claude Fable 5 --- LESSONS.md | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/LESSONS.md b/LESSONS.md index 10e3823..3de0d28 100644 --- a/LESSONS.md +++ b/LESSONS.md @@ -62,7 +62,15 @@ reproduction: [`RESULTS.md`](RESULTS.md). Everything on `google/gemma-4-E2B-it` 6. **Background jobs must be `setsid`'d** or the harness/session restart kills them mid-run. And `pkill -f ` will match your own launcher shell if the pattern appears in its command line. -7. **Zero-init adapter output layer ⇒ zero grads upstream at step 0** — on +7. **Sustained training in a closed cabinet = thermal hard-freezes.** Four + crashes in one day (journal stops mid-line, no OOM, no shutdown trace, + 37GB free at one death) on a DGX Spark that was stable all week under + light load. Pattern: only under hours of continuous GPU load; fixed by + opening the cabinet. Diagnose by exclusion: earlyoom quiet + journal + truncation + load correlation = thermal, not software. And do NOT + auto-restart training via @reboot cron on a thermally-suspect box — it + risks a crash loop with no human circuit breaker. +8. **Zero-init adapter output layer ⇒ zero grads upstream at step 0** — on `mlp[0]` this is expected (LoRA-B-style), not a bug; check the output layer's grad instead.