From 3477fb5da0adcabb510b03aa1d0674083fb62150 Mon Sep 17 00:00:00 2001 From: Nils Date: Wed, 15 Jul 2026 01:08:58 +0200 Subject: [PATCH] =?UTF-8?q?method:=20truncated-BPTT=20bias=20bound=20via?= =?UTF-8?q?=20rho(A)=20=E2=80=94=20contraction=20certifies=20tail-only=20g?= =?UTF-8?q?radients?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5 --- PAPER.md | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/PAPER.md b/PAPER.md index a6bd700..821b96a 100644 --- a/PAPER.md +++ b/PAPER.md @@ -145,6 +145,15 @@ adapter + entrance-faded LoRA rank 8 on the band's first layers, loop-only via a global toggle), and **stack** arms (distill-warm + loop training; distilled adapter evaluated in loop mode). +**Truncated backprop is certified by contraction.** Recurrent-regime arms +train with tail-only BPTT (gradients through the last 4 iterations; the +no-grad prefix stores no activations, so memory is constant in depth). +The truncation bias scales as ρ(A)^(T−tail) — at ρ=0.3 the discarded terms +are ≤1%, making the cheap estimator essentially exact; at ρ≥1 it is +dominated by what it discards. Stability, fixed-point convergence, valid +tail gradients, and the convergence-halting exit signal are all the same +dial. + **Inference.** Looped prompt states are causally independent of generated tokens: computed once at prefill, written into the KV cache by a hooked forward pass, generation native. Verified bit-identical to the slow path. @@ -405,6 +414,18 @@ Mixture-of-Recursions (2507.10524); Relaxed Recursive Transformers (2410.20672); Coconut; pause tokens (Goyal et al.) — whose trained variant proved a genuine rival, not a strawman (§3.2, §3.5). +**Saunshi et al. (2025)** argue looped transformers trade composition +against memorization: looping buys iterative reasoning, not fact storage. +Our results reproduce this axis *within one frozen model*: k>0 moves only +the plan-dependent (compositional) slice, leaves recall-flavored MC +benchmarks flat (§3.8), and the content-injecting distill arm — not the +loop — is what nudges knowledge benchmarks up. Their looping-based +regularization (loop harder on reasoning, relax for retrieval) has an +inference-time analogue in our difficulty gate: route predicted +plan-dependent prompts to k=4 and everything else to k=0, which is the +exact base model. Retrofit looping makes the composition/memorization +trade a *per-prompt routing decision* instead of a pretraining commitment. + What remains distinct here: interpretability-derived placement with causal validation; a fully frozen base with bit-exact k=0 and zero-decode-cost KV write-in; the complete attribution ladder including compute-matched