diff --git a/LESSONS.md b/LESSONS.md index 10e3823..3de0d28 100644 --- a/LESSONS.md +++ b/LESSONS.md @@ -62,7 +62,15 @@ reproduction: [`RESULTS.md`](RESULTS.md). Everything on `google/gemma-4-E2B-it` 6. **Background jobs must be `setsid`'d** or the harness/session restart kills them mid-run. And `pkill -f ` will match your own launcher shell if the pattern appears in its command line. -7. **Zero-init adapter output layer ⇒ zero grads upstream at step 0** — on +7. **Sustained training in a closed cabinet = thermal hard-freezes.** Four + crashes in one day (journal stops mid-line, no OOM, no shutdown trace, + 37GB free at one death) on a DGX Spark that was stable all week under + light load. Pattern: only under hours of continuous GPU load; fixed by + opening the cabinet. Diagnose by exclusion: earlyoom quiet + journal + truncation + load correlation = thermal, not software. And do NOT + auto-restart training via @reboot cron on a thermally-suspect box — it + risks a crash loop with no human circuit breaker. +8. **Zero-init adapter output layer ⇒ zero grads upstream at step 0** — on `mlp[0]` this is expected (LoRA-B-style), not a bug; check the output layer's grad instead.