item 28 pre-registered (Nils's variant): teacher-state distillation — frozen full-cot teacher's band-exit state at step end, cosine into burst s^10; λ amended 1.0→5.0 pre-run (smoke: baseline cos-dist 0.113)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -776,3 +776,38 @@ after it (job renamed zzz_t).
|
||||
(forces state-borne computation rather than KV re-reading) and
|
||||
anchor-at-prompt (iterations see the full settled question).
|
||||
Job: scripts/jobs/zzz_s_rungb_ii.sh.
|
||||
IN-FLIGHT NOTE (~02:20): lce 9.4 -> 2.9 by step 60 — the tape-free
|
||||
burst also encodes the step trajectory; encoding is never the
|
||||
obstacle. Accuracy pending.
|
||||
|
||||
28. **Teacher-state distillation into the burst (pre-registered
|
||||
2026-07-17 ~02:30, before running; Nils's variant: "meaningful CoT
|
||||
chunks yield internal state that we then teacher-force into the
|
||||
loop-only model").** Items 25/27 supervise the VERBAL SHADOW of
|
||||
the deleted step (token targets through the lens); this forces the
|
||||
FULL state. Teacher = the frozen warm-start (rung-A e400) adapter
|
||||
running the complete cot (deleted step visible), zero pauses, same
|
||||
carry architecture; capture its band-exit state at the deleted
|
||||
step's LAST token — "the state of having finished thinking the
|
||||
step" — one no-grad pass over the 413 parseable items at startup
|
||||
(targets fixed, no moving teacher). Student: identical zero-pause
|
||||
M=10 burst as item 27, but the loss is cosine distance between the
|
||||
burst's FINAL iterate s^10 and the teacher state, plus output CE.
|
||||
Weight AMENDED pre-run λ=1.0 -> 5.0 after the smoke measured the
|
||||
starting cosine distance at only 0.113 (nearby band-exit states
|
||||
share most structure); at 1.0 the term would be ~10x weaker than
|
||||
the CE and a null would be underpowered — 5x0.113 puts the two
|
||||
terms at comparable initial scale. No lens loss (one knob vs item 27: full-state
|
||||
targets instead of verbal-shadow targets; trajectory supervision
|
||||
dropped — only the endpoint is forced). Known approximation,
|
||||
stated in advance: teacher state is captured ~10 positions later
|
||||
in the sequence than the student anchor (RoPE position coloring);
|
||||
cosine + the carry machinery's routine state transplantation
|
||||
across nearby positions make this tolerable, but a null could
|
||||
partly hide here. Eval n=256: 0:0, 2:0 ii10 (matched), 2:0 ii0
|
||||
(ablation). Decision vs 31.6, same bands; the informative
|
||||
three-way is 25 (verbal, tape) vs 27 (verbal, no tape) vs 28
|
||||
(full state, no tape) — if 28 moves where 25/27 don't, the
|
||||
computation-carrying content lives OUTSIDE the verbalizable
|
||||
subspace; if none move, the read-side clamp test (item 29
|
||||
candidate) decides. Job: scripts/jobs/zzz_sa_rungb_ts.sh.
|
||||
|
||||
Reference in New Issue
Block a user