item 31 pre-registered (Nils's design): synthetic memory tokens — burst states → per-band-layer KV prefix via KVMemoryAdapter (in-place attention wrap, bit-exact disarmed, gated silent init); composed with frozen item-29 arm-1; +29 tf+fr in-flight note (30.5, FR term hurts)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -950,3 +950,35 @@ Nils's morning decision: clamp test vs pivot to the hybrid/A2 line
|
||||
matched accuracy; oracle meaningfully above 57.4 (early-stop
|
||||
rescues drift cases). Arm 2 (later): line-correctness /
|
||||
deferral head gating re-expansion. Job: scripts/jobs/zzz_w_metacog.sh.
|
||||
|
||||
31. **Synthetic memory tokens: per-layer KV prefix (pre-registered
|
||||
2026-07-17 ~15:50, before running; Nils's design: "the one read
|
||||
mechanism a frozen transformer natively possesses is attention over
|
||||
the KV cache... adapter maps L30 state -> KV entries at band
|
||||
layers"; injection variant chosen by Nils: per-layer KV prefix).**
|
||||
The read-side attack, composed with item 29: warm-start the arm-1
|
||||
adapter (its burst COMPUTES — trajectory-trained, ltf 0.085 — but
|
||||
is provably unread: ablation 8-8 p=1.0) and FREEZE it; train only
|
||||
a KVMemoryAdapter (6.3M params: shared code-512 trunk + per-band-
|
||||
layer k/v heads with per-layer geometry — gemma-4 mixes 256-d
|
||||
sliding and 512-d global heads) mapping the 10 burst iterates to
|
||||
post-RoPE (k,v) columns appended to every band layer's attention
|
||||
during the answer scan and at generation. Engineering, validated
|
||||
in smokes: the registered attention implementation is wrapped IN
|
||||
PLACE (config name untouched -> mask construction identical;
|
||||
disarmed = bit-exact, diff 0.0); zero-init value heads + learnable
|
||||
per-layer gate bias (init -10) make the memory silent at init
|
||||
(step-0 loss 0.108 = the arm-1 endpoint exactly); memory stays
|
||||
armed through backward (checkpoint recompute). Output CE only,
|
||||
200 steps, lr 1e-3. Eval: 0:0 sanity + 2:0 matched (burst +
|
||||
memory); the ablation is item 29's matched cell BY CONSTRUCTION
|
||||
(same frozen adapter and burst, memory absent) = 39.1. Decision:
|
||||
>44.1 (>=+5 over 39.1) = the consumption wall was a READ-PATH
|
||||
problem and native attention over state-derived KV breaches it;
|
||||
within +-5 = even natively readable computed states go unused ->
|
||||
the wall is not about the read mechanism either, and the hybrid/
|
||||
metacog lines carry the program. In-flight note on item 29 arm 2
|
||||
(tf+fr): matched 30.5 — the free-running term HURT (vs 39.1
|
||||
tf-only), reinforcing the training-signal attribution: clean TF
|
||||
transition gradients are the active ingredient.
|
||||
Job: scripts/jobs/zzz_x_kvmem.sh.
|
||||
|
||||
Reference in New Issue
Block a user