best-of-3 oracle/deployable split: realistic selector loses the overall edge (55.0/27.3 vs oracle 57.8/34.5); paper economics rewritten

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

PAPER.md

@@ -31,7 +31,8 @@ A complete attribution program then deflates the mechanism's mystique: the
 loop's content is *amortizable* (plan-distillation into the same adapter,
 recurrence-free, matches it; nothing stacks; looping distilled weights
 degrades them), width rivals depth (trained pause registers reach 36.4%),
-and compute-matched token baselines win overall accuracy outright. What
+and verifier-assisted (oracle) sampling wins overall accuracy at matched
+compute — though the *deployable* selector loses that edge entirely. What
 survives is precise: the implant owns exactly the plan-dependent slice at
 zero visible tokens and zero decode cost, transfers with the substrate
 rather than the task, and its placement is dictated by the lens. At 12B a
@@ -88,11 +89,14 @@ not laws.)*
    (parallel, prefill-shaped), zero decode overhead, bit-exact KV-cache
    write-in, k=0 recovers the base model exactly. Its competition at
    comparable compute (accounting in Appendix A — FLOPs, wall-clock, and
-   token budget do not rank the arms the same way): best-of-3 sampling wins
-   overall accuracy outright (57.2%); a
-   50-token visible plan ties the hard bucket. The value proposition is
-   *only*: no visible tokens, no decode latency, and the hard-slice
-   specialization (distill's 46% > budget-CoT's 40% > best-of-3's 33%).
+   token budget do not rank the arms the same way): *oracle* best-of-3
+   (verifier-assisted) wins overall accuracy (57.8%, paired p=0.020 vs
+   loop), but the *deployable* logprob-selected variant drops to 55.0%
+   overall / 27.3% hard — indistinguishable from the latent arms overall
+   and directionally behind on hard. A 50-token visible plan ties the hard
+   bucket. The value proposition without a verifier: no visible tokens, no
+   decode latency, and the hard-slice specialization (distill's 46% >
+   budget-CoT's 40% > deployable sampling's 27%).
 
 6. **Scale transfers only with a state-dependent stability dial.** At 12B the
    2B-tuned constant α=0.3 collapses overall accuracy (72.6%→43.0%); the
@@ -228,20 +232,26 @@ distinctive value is that it finds this content *without* plan supervision
 ### 3.4 Compute-matched honesty
 
 At approximately matched FLOPs (Appendix A gives the accounting, separated
-into FLOPs, wall-clock, and token budget), token-space baselines are strong:
-best-of-3 sampling wins overall accuracy against every latent arm (57.2%,
-CI [52.8, 61.5], vs loop 53.6 [49.2, 57.9] — point estimate higher, CIs
-overlap) by preserving easy items perfectly while sampling rescues some hard
-ones. A 50-token visible plan ties the loop's hard bucket. Both baselines
-are being rerun with per-item logs to enable paired tests against the latent
-arms; until those land, the overall-accuracy comparison rests on overlapping
-CIs and is stated as point-estimate-level. [BASELINES-PI PENDING]
-The latent implant's surviving advantages are qualitative: zero visible
-tokens (silent), zero decode overhead (prefill-parallel; sampling and CoT
-pay serially at bandwidth-bound decode), and the hard-slice crown under
-distillation (46% vs 40% budget-CoT vs 33% best-of-3). For deployment this
-means: the implant is a *latency/token-budget* technology with a side
-specialization in plan-dependent items — not an accuracy technology.
+into FLOPs, wall-clock, and token budget), the token-space comparison
+splits into two very different claims:
+
+| best-of-3 variant | overall | hard | vs loop (paired) |
+|---|---|---|---|
+| **oracle** (any-of-3 passes; needs a perfect verifier) | 57.8% | 34.5% | beats loop overall, p=0.020 |
+| **deployable** (highest mean logprob of 3) | 55.0% | 27.3% | n.s. overall (p=0.47); loop ahead on hard 16–7 (p=0.09) |
+
+The earlier draft's "sampling wins overall" was the **oracle** number — an
+upper bound requiring an external verifier that MBPP's own tests provide
+but a deployment does not. With the deployable selector (identical seeded
+samples, so the comparison is exact), best-of-3 is statistically
+indistinguishable from the latent arms overall, *behind* them
+directionally on the hard bucket, and pays ≈3× visible tokens and serial
+decode for it. Budget-CoT-50 remains the strongest honest token baseline
+(53.8% overall, hard 40.0%). The implant's advantages at matched compute
+without a verifier: zero visible tokens, zero decode overhead, and the
+hard-slice crown under distillation (46%). Where a task *does* come with a
+cheap verifier, oracle-style sampling is the better overall-accuracy
+spend — both halves belong in the deployment picture.
 
 ### 3.5 Width vs depth, and the task boundary