item 31 pre-registered (Nils's design): synthetic memory tokens — burst states → per-band-layer KV prefix via KVMemoryAdapter (in-place attention wrap, bit-exact disarmed, gated silent init); composed with frozen item-29 arm-1; +29 tf+fr in-flight note (30.5, FR term hurts)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Nils
2026-07-17 13:25:25 +02:00
co-authored by Claude Fable 5
parent 11604d0949
commit b18ffbd739
6 changed files with 231 additions and 11 deletions
+32
View File
@@ -950,3 +950,35 @@ Nils's morning decision: clamp test vs pivot to the hybrid/A2 line
matched accuracy; oracle meaningfully above 57.4 (early-stop
rescues drift cases). Arm 2 (later): line-correctness /
deferral head gating re-expansion. Job: scripts/jobs/zzz_w_metacog.sh.
31. **Synthetic memory tokens: per-layer KV prefix (pre-registered
2026-07-17 ~15:50, before running; Nils's design: "the one read
mechanism a frozen transformer natively possesses is attention over
the KV cache... adapter maps L30 state -> KV entries at band
layers"; injection variant chosen by Nils: per-layer KV prefix).**
The read-side attack, composed with item 29: warm-start the arm-1
adapter (its burst COMPUTES — trajectory-trained, ltf 0.085 — but
is provably unread: ablation 8-8 p=1.0) and FREEZE it; train only
a KVMemoryAdapter (6.3M params: shared code-512 trunk + per-band-
layer k/v heads with per-layer geometry — gemma-4 mixes 256-d
sliding and 512-d global heads) mapping the 10 burst iterates to
post-RoPE (k,v) columns appended to every band layer's attention
during the answer scan and at generation. Engineering, validated
in smokes: the registered attention implementation is wrapped IN
PLACE (config name untouched -> mask construction identical;
disarmed = bit-exact, diff 0.0); zero-init value heads + learnable
per-layer gate bias (init -10) make the memory silent at init
(step-0 loss 0.108 = the arm-1 endpoint exactly); memory stays
armed through backward (checkpoint recompute). Output CE only,
200 steps, lr 1e-3. Eval: 0:0 sanity + 2:0 matched (burst +
memory); the ablation is item 29's matched cell BY CONSTRUCTION
(same frozen adapter and burst, memory absent) = 39.1. Decision:
>44.1 (>=+5 over 39.1) = the consumption wall was a READ-PATH
problem and native attention over state-derived KV breaches it;
within +-5 = even natively readable computed states go unused ->
the wall is not about the read mechanism either, and the hybrid/
metacog lines carry the program. In-flight note on item 29 arm 2
(tf+fr): matched 30.5 — the free-running term HURT (vs 39.1
tf-only), reinforcing the training-signal attribution: clean TF
transition gradients are the active ingredient.
Job: scripts/jobs/zzz_x_kvmem.sh.