diff --git a/PAPER.md b/PAPER.md index e0aa1b0..e24df21 100644 --- a/PAPER.md +++ b/PAPER.md @@ -34,8 +34,12 @@ degrades them), width rivals depth (trained pause registers reach 36.4%), and verifier-assisted (oracle) sampling wins overall accuracy at matched compute — though the *deployable* selector loses that edge entirely. What survives is precise: the implant owns exactly the plan-dependent slice at -zero visible tokens and zero decode cost, transfers with the substrate -rather than the task, and its placement is dictated by the lens. At 12B a +zero visible tokens and zero decode cost, and its placement is dictated by +the lens. Transfer dissociates by distance: near-distribution the trained +and untrained implants tie (HumanEval); far from it (LiveCodeBench) the +*untrained* merge significantly helps while the trained content +significantly hurts — the learned content is task-local, the recurrence +substrate is general. At 12B a constant merge coefficient destroys the substrate; a state-dependent coefficient (3.8K parameters) restores MBPP but not Blocksworld or GSM8K — the anchor coefficient is the stability dial that unifies this work with @@ -174,8 +178,9 @@ hard 43.6%, overall 53.6%. L13: hard 17.9%, overall 34.4%. L9–L12: overall verified bit-identical) — the 12B model has no shared-KV layers, making it the unconfounded replication. Exit sweep at fixed entrance (L27/30/32/34): hard 39.3–46.4%, within seed spread. The lens boundary is necessary; the -exit is a free parameter. (The L23-exit arm died in training; a rerun is in -progress — the four completed exits bracket it. [L23 PENDING]) +exit is a free parameter — the completed five-point exit sweep +(L23/27/30/32/34) spans hard 39.3–46.4% with L23 at the top (46.4% at k=2), +all within seed spread. ### 3.2 The attribution ladder @@ -297,7 +302,28 @@ significant** (paired McNemar at k=2, 9 vs 7 discordant, p=0.80). What transfers significantly is the *merge perturbation itself*, not the MBPP-trained content — the cleanest evidence that off-distribution value is substrate-shaped rather than task-memorized. (The transferred pause adapter -reaches 66.5%, hard 38.9%, consistent with the same reading.) **Rust/MultiPL-E** (Python-trained, different +reaches 66.5%, hard 38.9%, consistent with the same reading.) + +**LiveCodeBench sharpens this into a dissociation** (150 newest stdin +problems, Nov 2024–Apr 2025, execution-verified; no LCB training anywhere +in the pipeline; base 18.7%): + +| arm (MBPP-trained where trained) | overall | hard (n=25) | vs base, paired | +|---|---|---|---| +| **untrained merge, k=4** | **24.0%** | **36.0%** | **+**, p=0.039 | +| trained loop, k=4 | 15.3% | 8.0% | −, p=0.23 | +| distill FF, k=1 | 12.7% | 16.0% | **−**, p=0.049 | + +Far from distribution, the *trained content is a liability* (distill +significantly hurts; untrained-vs-trained-loop is 14–1 discordant, +p=0.001) while the *untrained anchored recurrence significantly helps* — +the training-free regime of Lys et al. is the right choice off-distribution, +and the amortized-content reading of §3.3 predicts exactly this: what the +adapter learned is MBPP-shaped plan content, valuable where plans look like +MBPP plans and harmful where they don't. Transfer ordering by distance: +HumanEval (near) — trained ≈ untrained; Rust (mid) — trained helps the hard +bucket; LCB (far) — untrained wins outright. Caveats: single seed per arm, +hard n=25, one benchmark at the far end. **Rust/MultiPL-E** (Python-trained, different language, compile-run-verified): hard 8.0%→24.0% (p=0.125 at n=25 — directionally consistent, underpowered). **Blocksworld** MBPP-transfer: hard 0→14.3% (task-trained: 43%). Content transfers where the substrate's @@ -316,9 +342,22 @@ removing the easy-item perturbation tax. k=0 is the exact base model by construction — the implant is removable at token granularity. **General-capability panel** (ARC-Challenge, WinoGrande, HellaSwag, MMLU; -length-normalized MC scoring with the loop applied to the context span) is -running on the Spark; results will quantify what k>0 does to off-task -abilities. [PENDING — fill on completion.] +800 items each, length-normalized MC likelihood via the chat template, loop +applied to the context span). The safety answer is clean — **k>0 does not +damage general abilities**: + +| arm | ARC-C | WinoGrande | HellaSwag | MMLU | +|---|---|---|---|---| +| base (k=0) | 36.0 | 55.9 | 52.3 | 30.1 | +| loop k=2 (MBPP adapter) | 36.1 | 55.3 | 49.6 | 31.3 | +| distill FF (MBPP) | 41.8 | 56.6 | 57.0 | 31.8 | + +The loop arm is flat within noise (largest move −2.6 on HellaSwag, +unpaired n=800). The distill adapter *nominally improves* every benchmark +(+5.8 ARC, +4.8 HellaSwag) — consistent with §3.7's finding that these +implants carry a generically useful perturbation component, though +MC-likelihood scoring and generation quality are different regimes (see +the LCB result below before reading this as free capability). ### 3.9 Negative results with content @@ -374,9 +413,15 @@ are small (n=55/38/25); within-ladder orderings are not individually significant, and only the pooled hard effect and the HumanEval overall gain survive multiple-comparison scrutiny. Bucket membership derives from greedy labeling runs (consensus-k0 robustness check moves numbers <2 points, but -both checks share the base model; an independent difficulty proxy is an -open external check). A third architecture family was not run; -LiveCodeBench (contamination-safe) was not run; rung-2 was not run at 12B. The easy-item perturbation tax persists wherever the +both checks share the base model; an independent 12B-relabeling proxy is +running). A third architecture family was not run; rung-2 was not run at +12B. LiveCodeBench: single seed per arm, hard n=25, stdin-judged problems +only, and its newest shard (Apr 2025) is *newer than MBPP by years* but +not provably past the base model's undisclosed training cutoff — we claim +recency, not proven non-contamination. The capability panel is +MC-likelihood, not generation; its "no damage" answer does not extend to +generation quality off-distribution (LCB shows trained arms *do* hurt +there). The easy-item perturbation tax persists wherever the gate's precision fails. MBPP/GSM8K likely overlap pretraining data; both arms share contamination, and memorized items land in the easy bucket, but bucket composition is contamination-sensitive. The capability panel diff --git a/results-loop/eval_code_tap23_rerun.json b/results-loop/eval_code_tap23_rerun.json new file mode 100644 index 0000000..a3a06c9 --- /dev/null +++ b/results-loop/eval_code_tap23_rerun.json @@ -0,0 +1,3036 @@ +{ + "tag": "tap23_rerun", + "ks": { + "0": { + "acc": 0.492, + "by_label": { + "easy": 0.9836065573770492, + "hard": 0.03571428571428571, + "drop": 0.02 + }, + "per_item": [ + { + "task_id": 11, + "ok": true + }, + { + "task_id": 12, + "ok": false + }, + { + "task_id": 13, + "ok": false + }, + { + "task_id": 14, + "ok": true + }, + { + "task_id": 15, + "ok": false + }, + { + "task_id": 16, + "ok": false + }, + { + "task_id": 17, + "ok": true + }, + { + "task_id": 18, + "ok": true + }, + { + "task_id": 19, + "ok": true + }, + { + "task_id": 20, + "ok": false + }, + { + "task_id": 21, + "ok": false + }, + { + "task_id": 22, + "ok": true + }, + { + "task_id": 23, + "ok": true + }, + { + "task_id": 24, + "ok": false + }, + { + "task_id": 25, + "ok": true + }, + { + "task_id": 26, + "ok": false + }, + { + "task_id": 27, + "ok": true + }, + { + "task_id": 28, + "ok": true + }, + { + "task_id": 29, + "ok": true + }, + { + "task_id": 30, + "ok": true + }, + { + "task_id": 31, + "ok": false + }, + { + "task_id": 32, + "ok": true + }, + { + "task_id": 33, + "ok": false + }, + { + "task_id": 34, + "ok": false + }, + { + "task_id": 35, + "ok": false + }, + { + "task_id": 36, + "ok": false + }, + { + "task_id": 37, + "ok": true + }, + { + "task_id": 38, + "ok": false + }, + { + "task_id": 39, + "ok": false + }, + { + "task_id": 40, + "ok": true + }, + { + "task_id": 41, + "ok": true + }, + { + "task_id": 42, + "ok": false + }, + { + "task_id": 43, + "ok": false + }, + { + "task_id": 44, + "ok": true + }, + { + "task_id": 45, + "ok": true + }, + { + "task_id": 46, + "ok": true + }, + { + "task_id": 47, + "ok": false + }, + { + "task_id": 48, + "ok": false + }, + { + "task_id": 49, + "ok": true + }, + { + "task_id": 50, + "ok": true + }, + { + "task_id": 51, + "ok": true + }, + { + "task_id": 52, + "ok": true + }, + { + "task_id": 53, + "ok": true + }, + { + "task_id": 54, + "ok": false + }, + { + "task_id": 55, + "ok": true + }, + { + "task_id": 56, + "ok": true + }, + { + "task_id": 57, + "ok": true + }, + { + "task_id": 58, + "ok": true + }, + { + "task_id": 59, + "ok": false + }, + { + "task_id": 60, + "ok": false + }, + { + "task_id": 61, + "ok": true + }, + { + "task_id": 62, + "ok": true + }, + { + "task_id": 63, + "ok": true + }, + { + "task_id": 64, + "ok": true + }, + { + "task_id": 65, + "ok": true + }, + { + "task_id": 66, + "ok": true + }, + { + "task_id": 67, + "ok": false + }, + { + "task_id": 68, + "ok": true + }, + { + "task_id": 69, + "ok": true + }, + { + "task_id": 70, + "ok": true + }, + { + "task_id": 71, + "ok": false + }, + { + "task_id": 72, + "ok": false + }, + { + "task_id": 73, + "ok": true + }, + { + "task_id": 74, + "ok": false + }, + { + "task_id": 75, + "ok": true + }, + { + "task_id": 76, + "ok": true + }, + { + "task_id": 77, + "ok": false + }, + { + "task_id": 78, + "ok": true + }, + { + "task_id": 79, + "ok": true + }, + { + "task_id": 80, + "ok": true + }, + { + "task_id": 81, + "ok": false + }, + { + "task_id": 82, + "ok": true + }, + { + "task_id": 83, + "ok": false + }, + { + "task_id": 84, + "ok": false + }, + { + "task_id": 85, + "ok": true + }, + { + "task_id": 86, + "ok": true + }, + { + "task_id": 87, + "ok": false + }, + { + "task_id": 88, + "ok": true + }, + { + "task_id": 89, + "ok": true + }, + { + "task_id": 90, + "ok": true + }, + { + "task_id": 91, + "ok": true + }, + { + "task_id": 92, + "ok": false + }, + { + "task_id": 93, + "ok": true + }, + { + "task_id": 94, + "ok": true + }, + { + "task_id": 95, + "ok": false + }, + { + "task_id": 96, + "ok": true + }, + { + "task_id": 97, + "ok": true + }, + { + "task_id": 98, + "ok": true + }, + { + "task_id": 99, + "ok": true + }, + { + "task_id": 100, + "ok": false + }, + { + "task_id": 101, + "ok": false + }, + { + "task_id": 102, + "ok": true + }, + { + "task_id": 103, + "ok": false + }, + { + "task_id": 104, + "ok": true + }, + { + "task_id": 105, + "ok": true + }, + { + "task_id": 106, + "ok": false + }, + { + "task_id": 107, + "ok": false + }, + { + "task_id": 108, + "ok": false + }, + { + "task_id": 109, + "ok": false + }, + { + "task_id": 110, + "ok": false + }, + { + "task_id": 111, + "ok": false + }, + { + "task_id": 112, + "ok": false + }, + { + "task_id": 113, + "ok": true + }, + { + "task_id": 114, + "ok": false + }, + { + "task_id": 115, + "ok": true + }, + { + "task_id": 116, + "ok": true + }, + { + "task_id": 117, + "ok": true + }, + { + "task_id": 118, + "ok": false + }, + { + "task_id": 119, + "ok": false + }, + { + "task_id": 120, + "ok": false + }, + { + "task_id": 121, + "ok": false + }, + { + "task_id": 122, + "ok": false + }, + { + "task_id": 123, + "ok": true + }, + { + "task_id": 124, + "ok": false + }, + { + "task_id": 125, + "ok": false + }, + { + "task_id": 126, + "ok": false + }, + { + "task_id": 127, + "ok": true + }, + { + "task_id": 128, + "ok": true + }, + { + "task_id": 129, + "ok": false + }, + { + "task_id": 130, + "ok": false + }, + { + "task_id": 131, + "ok": false + }, + { + "task_id": 132, + "ok": true + }, + { + "task_id": 133, + "ok": true + }, + { + "task_id": 134, + "ok": false + }, + { + "task_id": 135, + "ok": true + }, + { + "task_id": 136, + "ok": false + }, + { + "task_id": 137, + "ok": false + }, + { + "task_id": 138, + "ok": false + }, + { + "task_id": 139, + "ok": false + }, + { + "task_id": 140, + "ok": false + }, + { + "task_id": 141, + "ok": false + }, + { + "task_id": 142, + "ok": false + }, + { + "task_id": 143, + "ok": false + }, + { + "task_id": 144, + "ok": false + }, + { + "task_id": 145, + "ok": true + }, + { + "task_id": 146, + "ok": false + }, + { + "task_id": 147, + "ok": false + }, + { + "task_id": 148, + "ok": false + }, + { + "task_id": 149, + "ok": false + }, + { + "task_id": 150, + "ok": false + }, + { + "task_id": 151, + "ok": true + }, + { + "task_id": 152, + "ok": false + }, + { + "task_id": 153, + "ok": true + }, + { + "task_id": 154, + "ok": true + }, + { + "task_id": 155, + "ok": false + }, + { + "task_id": 156, + "ok": true + }, + { + "task_id": 157, + "ok": true + }, + { + "task_id": 158, + "ok": false + }, + { + "task_id": 159, + "ok": false + }, + { + "task_id": 160, + "ok": false + }, + { + "task_id": 161, + "ok": true + }, + { + "task_id": 162, + "ok": true + }, + { + "task_id": 163, + "ok": true + }, + { + "task_id": 164, + "ok": false + }, + { + "task_id": 165, + "ok": false + }, + { + "task_id": 166, + "ok": false + }, + { + "task_id": 167, + "ok": true + }, + { + "task_id": 168, + "ok": true + }, + { + "task_id": 169, + "ok": false + }, + { + "task_id": 170, + "ok": false + }, + { + "task_id": 171, + "ok": true + }, + { + "task_id": 172, + "ok": true + }, + { + "task_id": 173, + "ok": true + }, + { + "task_id": 174, + "ok": true + }, + { + "task_id": 175, + "ok": true + }, + { + "task_id": 176, + "ok": true + }, + { + "task_id": 177, + "ok": false + }, + { + "task_id": 178, + "ok": true + }, + { + "task_id": 179, + "ok": true + }, + { + "task_id": 180, + "ok": false + }, + { + "task_id": 181, + "ok": false + }, + { + "task_id": 182, + "ok": false + }, + { + "task_id": 183, + "ok": false + }, + { + "task_id": 184, + "ok": false + }, + { + "task_id": 185, + "ok": false + }, + { + "task_id": 186, + "ok": true + }, + { + "task_id": 187, + "ok": false + }, + { + "task_id": 188, + "ok": false + }, + { + "task_id": 189, + "ok": false + }, + { + "task_id": 190, + "ok": false + }, + { + "task_id": 191, + "ok": true + }, + { + "task_id": 192, + "ok": true + }, + { + "task_id": 193, + "ok": false + }, + { + "task_id": 194, + "ok": true + }, + { + "task_id": 195, + "ok": false + }, + { + "task_id": 196, + "ok": true + }, + { + "task_id": 197, + "ok": true + }, + { + "task_id": 198, + "ok": false + }, + { + "task_id": 199, + "ok": true + }, + { + "task_id": 200, + "ok": true + }, + { + "task_id": 201, + "ok": true + }, + { + "task_id": 202, + "ok": true + }, + { + "task_id": 203, + "ok": true + }, + { + "task_id": 204, + "ok": true + }, + { + "task_id": 205, + "ok": false + }, + { + "task_id": 206, + "ok": false + }, + { + "task_id": 207, + "ok": false + }, + { + "task_id": 208, + "ok": false + }, + { + "task_id": 209, + "ok": false + }, + { + "task_id": 210, + "ok": true + }, + { + "task_id": 211, + "ok": false + }, + { + "task_id": 212, + "ok": false + }, + { + "task_id": 213, + "ok": true + }, + { + "task_id": 214, + "ok": true + }, + { + "task_id": 215, + "ok": false + }, + { + "task_id": 216, + "ok": false + }, + { + "task_id": 217, + "ok": true + }, + { + "task_id": 218, + "ok": false + }, + { + "task_id": 219, + "ok": false + }, + { + "task_id": 220, + "ok": false + }, + { + "task_id": 221, + "ok": true + }, + { + "task_id": 222, + "ok": true + }, + { + "task_id": 223, + "ok": false + }, + { + "task_id": 224, + "ok": true + }, + { + "task_id": 225, + "ok": false + }, + { + "task_id": 226, + "ok": true + }, + { + "task_id": 227, + "ok": true + }, + { + "task_id": 228, + "ok": false + }, + { + "task_id": 229, + "ok": false + }, + { + "task_id": 230, + "ok": true + }, + { + "task_id": 231, + "ok": false + }, + { + "task_id": 232, + "ok": true + }, + { + "task_id": 233, + "ok": false + }, + { + "task_id": 234, + "ok": true + }, + { + "task_id": 235, + "ok": false + }, + { + "task_id": 236, + "ok": false + }, + { + "task_id": 237, + "ok": false + }, + { + "task_id": 238, + "ok": true + }, + { + "task_id": 239, + "ok": false + }, + { + "task_id": 240, + "ok": true + }, + { + "task_id": 241, + "ok": false + }, + { + "task_id": 242, + "ok": true + }, + { + "task_id": 243, + "ok": false + }, + { + "task_id": 244, + "ok": true + }, + { + "task_id": 245, + "ok": false + }, + { + "task_id": 246, + "ok": true + }, + { + "task_id": 247, + "ok": false + }, + { + "task_id": 248, + "ok": false + }, + { + "task_id": 249, + "ok": false + }, + { + "task_id": 250, + "ok": true + }, + { + "task_id": 251, + "ok": true + }, + { + "task_id": 252, + "ok": true + }, + { + "task_id": 253, + "ok": true + }, + { + "task_id": 254, + "ok": true + }, + { + "task_id": 255, + "ok": false + }, + { + "task_id": 256, + "ok": false + }, + { + "task_id": 257, + "ok": true + }, + { + "task_id": 258, + "ok": false + }, + { + "task_id": 259, + "ok": false + }, + { + "task_id": 260, + "ok": false + } + ] + }, + "2": { + "acc": 0.532, + "by_label": { + "easy": 0.9098360655737705, + "hard": 0.4642857142857143, + "drop": 0.09 + }, + "per_item": [ + { + "task_id": 11, + "ok": true + }, + { + "task_id": 12, + "ok": true + }, + { + "task_id": 13, + "ok": false + }, + { + "task_id": 14, + "ok": true + }, + { + "task_id": 15, + "ok": false + }, + { + "task_id": 16, + "ok": true + }, + { + "task_id": 17, + "ok": true + }, + { + "task_id": 18, + "ok": true + }, + { + "task_id": 19, + "ok": true + }, + { + "task_id": 20, + "ok": false + }, + { + "task_id": 21, + "ok": false + }, + { + "task_id": 22, + "ok": true + }, + { + "task_id": 23, + "ok": true + }, + { + "task_id": 24, + "ok": false + }, + { + "task_id": 25, + "ok": true + }, + { + "task_id": 26, + "ok": false + }, + { + "task_id": 27, + "ok": true + }, + { + "task_id": 28, + "ok": true + }, + { + "task_id": 29, + "ok": true + }, + { + "task_id": 30, + "ok": true + }, + { + "task_id": 31, + "ok": false + }, + { + "task_id": 32, + "ok": true + }, + { + "task_id": 33, + "ok": false + }, + { + "task_id": 34, + "ok": false + }, + { + "task_id": 35, + "ok": false + }, + { + "task_id": 36, + "ok": false + }, + { + "task_id": 37, + "ok": true + }, + { + "task_id": 38, + "ok": false + }, + { + "task_id": 39, + "ok": false + }, + { + "task_id": 40, + "ok": true + }, + { + "task_id": 41, + "ok": true + }, + { + "task_id": 42, + "ok": false + }, + { + "task_id": 43, + "ok": false + }, + { + "task_id": 44, + "ok": false + }, + { + "task_id": 45, + "ok": true + }, + { + "task_id": 46, + "ok": true + }, + { + "task_id": 47, + "ok": false + }, + { + "task_id": 48, + "ok": false + }, + { + "task_id": 49, + "ok": true + }, + { + "task_id": 50, + "ok": true + }, + { + "task_id": 51, + "ok": true + }, + { + "task_id": 52, + "ok": true + }, + { + "task_id": 53, + "ok": true + }, + { + "task_id": 54, + "ok": true + }, + { + "task_id": 55, + "ok": true + }, + { + "task_id": 56, + "ok": true + }, + { + "task_id": 57, + "ok": false + }, + { + "task_id": 58, + "ok": true + }, + { + "task_id": 59, + "ok": false + }, + { + "task_id": 60, + "ok": false + }, + { + "task_id": 61, + "ok": true + }, + { + "task_id": 62, + "ok": true + }, + { + "task_id": 63, + "ok": true + }, + { + "task_id": 64, + "ok": true + }, + { + "task_id": 65, + "ok": true + }, + { + "task_id": 66, + "ok": true + }, + { + "task_id": 67, + "ok": false + }, + { + "task_id": 68, + "ok": true + }, + { + "task_id": 69, + "ok": true + }, + { + "task_id": 70, + "ok": true + }, + { + "task_id": 71, + "ok": false + }, + { + "task_id": 72, + "ok": true + }, + { + "task_id": 73, + "ok": false + }, + { + "task_id": 74, + "ok": false + }, + { + "task_id": 75, + "ok": true + }, + { + "task_id": 76, + "ok": true + }, + { + "task_id": 77, + "ok": false + }, + { + "task_id": 78, + "ok": true + }, + { + "task_id": 79, + "ok": true + }, + { + "task_id": 80, + "ok": true + }, + { + "task_id": 81, + "ok": false + }, + { + "task_id": 82, + "ok": true + }, + { + "task_id": 83, + "ok": false + }, + { + "task_id": 84, + "ok": false + }, + { + "task_id": 85, + "ok": true + }, + { + "task_id": 86, + "ok": true + }, + { + "task_id": 87, + "ok": false + }, + { + "task_id": 88, + "ok": true + }, + { + "task_id": 89, + "ok": true + }, + { + "task_id": 90, + "ok": true + }, + { + "task_id": 91, + "ok": true + }, + { + "task_id": 92, + "ok": false + }, + { + "task_id": 93, + "ok": true + }, + { + "task_id": 94, + "ok": true + }, + { + "task_id": 95, + "ok": true + }, + { + "task_id": 96, + "ok": true + }, + { + "task_id": 97, + "ok": true + }, + { + "task_id": 98, + "ok": true + }, + { + "task_id": 99, + "ok": true + }, + { + "task_id": 100, + "ok": false + }, + { + "task_id": 101, + "ok": false + }, + { + "task_id": 102, + "ok": true + }, + { + "task_id": 103, + "ok": false + }, + { + "task_id": 104, + "ok": true + }, + { + "task_id": 105, + "ok": false + }, + { + "task_id": 106, + "ok": false + }, + { + "task_id": 107, + "ok": false + }, + { + "task_id": 108, + "ok": false + }, + { + "task_id": 109, + "ok": true + }, + { + "task_id": 110, + "ok": false + }, + { + "task_id": 111, + "ok": false + }, + { + "task_id": 112, + "ok": false + }, + { + "task_id": 113, + "ok": false + }, + { + "task_id": 114, + "ok": false + }, + { + "task_id": 115, + "ok": true + }, + { + "task_id": 116, + "ok": true + }, + { + "task_id": 117, + "ok": true + }, + { + "task_id": 118, + "ok": true + }, + { + "task_id": 119, + "ok": false + }, + { + "task_id": 120, + "ok": true + }, + { + "task_id": 121, + "ok": false + }, + { + "task_id": 122, + "ok": false + }, + { + "task_id": 123, + "ok": false + }, + { + "task_id": 124, + "ok": false + }, + { + "task_id": 125, + "ok": false + }, + { + "task_id": 126, + "ok": false + }, + { + "task_id": 127, + "ok": true + }, + { + "task_id": 128, + "ok": true + }, + { + "task_id": 129, + "ok": false + }, + { + "task_id": 130, + "ok": true + }, + { + "task_id": 131, + "ok": true + }, + { + "task_id": 132, + "ok": true + }, + { + "task_id": 133, + "ok": true + }, + { + "task_id": 134, + "ok": false + }, + { + "task_id": 135, + "ok": true + }, + { + "task_id": 136, + "ok": false + }, + { + "task_id": 137, + "ok": false + }, + { + "task_id": 138, + "ok": false + }, + { + "task_id": 139, + "ok": false + }, + { + "task_id": 140, + "ok": false + }, + { + "task_id": 141, + "ok": false + }, + { + "task_id": 142, + "ok": false + }, + { + "task_id": 143, + "ok": false + }, + { + "task_id": 144, + "ok": false + }, + { + "task_id": 145, + "ok": true + }, + { + "task_id": 146, + "ok": false + }, + { + "task_id": 147, + "ok": false + }, + { + "task_id": 148, + "ok": false + }, + { + "task_id": 149, + "ok": true + }, + { + "task_id": 150, + "ok": false + }, + { + "task_id": 151, + "ok": true + }, + { + "task_id": 152, + "ok": false + }, + { + "task_id": 153, + "ok": true + }, + { + "task_id": 154, + "ok": true + }, + { + "task_id": 155, + "ok": false + }, + { + "task_id": 156, + "ok": true + }, + { + "task_id": 157, + "ok": true + }, + { + "task_id": 158, + "ok": false + }, + { + "task_id": 159, + "ok": true + }, + { + "task_id": 160, + "ok": false + }, + { + "task_id": 161, + "ok": true + }, + { + "task_id": 162, + "ok": true + }, + { + "task_id": 163, + "ok": true + }, + { + "task_id": 164, + "ok": false + }, + { + "task_id": 165, + "ok": true + }, + { + "task_id": 166, + "ok": true + }, + { + "task_id": 167, + "ok": true + }, + { + "task_id": 168, + "ok": true + }, + { + "task_id": 169, + "ok": false + }, + { + "task_id": 170, + "ok": true + }, + { + "task_id": 171, + "ok": true + }, + { + "task_id": 172, + "ok": false + }, + { + "task_id": 173, + "ok": true + }, + { + "task_id": 174, + "ok": true + }, + { + "task_id": 175, + "ok": true + }, + { + "task_id": 176, + "ok": true + }, + { + "task_id": 177, + "ok": false + }, + { + "task_id": 178, + "ok": true + }, + { + "task_id": 179, + "ok": true + }, + { + "task_id": 180, + "ok": false + }, + { + "task_id": 181, + "ok": false + }, + { + "task_id": 182, + "ok": false + }, + { + "task_id": 183, + "ok": false + }, + { + "task_id": 184, + "ok": false + }, + { + "task_id": 185, + "ok": false + }, + { + "task_id": 186, + "ok": true + }, + { + "task_id": 187, + "ok": true + }, + { + "task_id": 188, + "ok": false + }, + { + "task_id": 189, + "ok": false + }, + { + "task_id": 190, + "ok": false + }, + { + "task_id": 191, + "ok": true + }, + { + "task_id": 192, + "ok": true + }, + { + "task_id": 193, + "ok": true + }, + { + "task_id": 194, + "ok": false + }, + { + "task_id": 195, + "ok": true + }, + { + "task_id": 196, + "ok": true + }, + { + "task_id": 197, + "ok": true + }, + { + "task_id": 198, + "ok": false + }, + { + "task_id": 199, + "ok": true + }, + { + "task_id": 200, + "ok": true + }, + { + "task_id": 201, + "ok": true + }, + { + "task_id": 202, + "ok": true + }, + { + "task_id": 203, + "ok": true + }, + { + "task_id": 204, + "ok": true + }, + { + "task_id": 205, + "ok": false + }, + { + "task_id": 206, + "ok": false + }, + { + "task_id": 207, + "ok": false + }, + { + "task_id": 208, + "ok": false + }, + { + "task_id": 209, + "ok": false + }, + { + "task_id": 210, + "ok": true + }, + { + "task_id": 211, + "ok": false + }, + { + "task_id": 212, + "ok": true + }, + { + "task_id": 213, + "ok": false + }, + { + "task_id": 214, + "ok": true + }, + { + "task_id": 215, + "ok": false + }, + { + "task_id": 216, + "ok": false + }, + { + "task_id": 217, + "ok": false + }, + { + "task_id": 218, + "ok": false + }, + { + "task_id": 219, + "ok": false + }, + { + "task_id": 220, + "ok": false + }, + { + "task_id": 221, + "ok": true + }, + { + "task_id": 222, + "ok": true + }, + { + "task_id": 223, + "ok": false + }, + { + "task_id": 224, + "ok": true + }, + { + "task_id": 225, + "ok": true + }, + { + "task_id": 226, + "ok": true + }, + { + "task_id": 227, + "ok": true + }, + { + "task_id": 228, + "ok": false + }, + { + "task_id": 229, + "ok": false + }, + { + "task_id": 230, + "ok": false + }, + { + "task_id": 231, + "ok": true + }, + { + "task_id": 232, + "ok": true + }, + { + "task_id": 233, + "ok": false + }, + { + "task_id": 234, + "ok": true + }, + { + "task_id": 235, + "ok": false + }, + { + "task_id": 236, + "ok": false + }, + { + "task_id": 237, + "ok": false + }, + { + "task_id": 238, + "ok": true + }, + { + "task_id": 239, + "ok": false + }, + { + "task_id": 240, + "ok": true + }, + { + "task_id": 241, + "ok": false + }, + { + "task_id": 242, + "ok": true + }, + { + "task_id": 243, + "ok": false + }, + { + "task_id": 244, + "ok": true + }, + { + "task_id": 245, + "ok": false + }, + { + "task_id": 246, + "ok": false + }, + { + "task_id": 247, + "ok": false + }, + { + "task_id": 248, + "ok": false + }, + { + "task_id": 249, + "ok": true + }, + { + "task_id": 250, + "ok": true + }, + { + "task_id": 251, + "ok": true + }, + { + "task_id": 252, + "ok": true + }, + { + "task_id": 253, + "ok": true + }, + { + "task_id": 254, + "ok": false + }, + { + "task_id": 255, + "ok": false + }, + { + "task_id": 256, + "ok": false + }, + { + "task_id": 257, + "ok": true + }, + { + "task_id": 258, + "ok": true + }, + { + "task_id": 259, + "ok": false + }, + { + "task_id": 260, + "ok": false + } + ] + }, + "4": { + "acc": 0.524, + "by_label": { + "easy": 0.8934426229508197, + "hard": 0.42857142857142855, + "drop": 0.1 + }, + "per_item": [ + { + "task_id": 11, + "ok": true + }, + { + "task_id": 12, + "ok": true + }, + { + "task_id": 13, + "ok": false + }, + { + "task_id": 14, + "ok": true + }, + { + "task_id": 15, + "ok": false + }, + { + "task_id": 16, + "ok": false + }, + { + "task_id": 17, + "ok": true + }, + { + "task_id": 18, + "ok": true + }, + { + "task_id": 19, + "ok": true + }, + { + "task_id": 20, + "ok": false + }, + { + "task_id": 21, + "ok": false + }, + { + "task_id": 22, + "ok": true + }, + { + "task_id": 23, + "ok": true + }, + { + "task_id": 24, + "ok": false + }, + { + "task_id": 25, + "ok": true + }, + { + "task_id": 26, + "ok": false + }, + { + "task_id": 27, + "ok": true + }, + { + "task_id": 28, + "ok": true + }, + { + "task_id": 29, + "ok": true + }, + { + "task_id": 30, + "ok": true + }, + { + "task_id": 31, + "ok": false + }, + { + "task_id": 32, + "ok": true + }, + { + "task_id": 33, + "ok": false + }, + { + "task_id": 34, + "ok": false + }, + { + "task_id": 35, + "ok": true + }, + { + "task_id": 36, + "ok": false + }, + { + "task_id": 37, + "ok": true + }, + { + "task_id": 38, + "ok": false + }, + { + "task_id": 39, + "ok": false + }, + { + "task_id": 40, + "ok": true + }, + { + "task_id": 41, + "ok": true + }, + { + "task_id": 42, + "ok": false + }, + { + "task_id": 43, + "ok": false + }, + { + "task_id": 44, + "ok": false + }, + { + "task_id": 45, + "ok": true + }, + { + "task_id": 46, + "ok": true + }, + { + "task_id": 47, + "ok": false + }, + { + "task_id": 48, + "ok": false + }, + { + "task_id": 49, + "ok": true + }, + { + "task_id": 50, + "ok": false + }, + { + "task_id": 51, + "ok": true + }, + { + "task_id": 52, + "ok": true + }, + { + "task_id": 53, + "ok": true + }, + { + "task_id": 54, + "ok": true + }, + { + "task_id": 55, + "ok": true + }, + { + "task_id": 56, + "ok": true + }, + { + "task_id": 57, + "ok": true + }, + { + "task_id": 58, + "ok": true + }, + { + "task_id": 59, + "ok": false + }, + { + "task_id": 60, + "ok": true + }, + { + "task_id": 61, + "ok": true + }, + { + "task_id": 62, + "ok": true + }, + { + "task_id": 63, + "ok": false + }, + { + "task_id": 64, + "ok": true + }, + { + "task_id": 65, + "ok": true + }, + { + "task_id": 66, + "ok": true + }, + { + "task_id": 67, + "ok": false + }, + { + "task_id": 68, + "ok": true + }, + { + "task_id": 69, + "ok": true + }, + { + "task_id": 70, + "ok": true + }, + { + "task_id": 71, + "ok": false + }, + { + "task_id": 72, + "ok": false + }, + { + "task_id": 73, + "ok": false + }, + { + "task_id": 74, + "ok": false + }, + { + "task_id": 75, + "ok": true + }, + { + "task_id": 76, + "ok": true + }, + { + "task_id": 77, + "ok": false + }, + { + "task_id": 78, + "ok": true + }, + { + "task_id": 79, + "ok": true + }, + { + "task_id": 80, + "ok": true + }, + { + "task_id": 81, + "ok": false + }, + { + "task_id": 82, + "ok": true + }, + { + "task_id": 83, + "ok": false + }, + { + "task_id": 84, + "ok": false + }, + { + "task_id": 85, + "ok": true + }, + { + "task_id": 86, + "ok": true + }, + { + "task_id": 87, + "ok": false + }, + { + "task_id": 88, + "ok": true + }, + { + "task_id": 89, + "ok": false + }, + { + "task_id": 90, + "ok": true + }, + { + "task_id": 91, + "ok": true + }, + { + "task_id": 92, + "ok": false + }, + { + "task_id": 93, + "ok": true + }, + { + "task_id": 94, + "ok": true + }, + { + "task_id": 95, + "ok": true + }, + { + "task_id": 96, + "ok": true + }, + { + "task_id": 97, + "ok": true + }, + { + "task_id": 98, + "ok": true + }, + { + "task_id": 99, + "ok": true + }, + { + "task_id": 100, + "ok": false + }, + { + "task_id": 101, + "ok": false + }, + { + "task_id": 102, + "ok": true + }, + { + "task_id": 103, + "ok": false + }, + { + "task_id": 104, + "ok": true + }, + { + "task_id": 105, + "ok": true + }, + { + "task_id": 106, + "ok": false + }, + { + "task_id": 107, + "ok": false + }, + { + "task_id": 108, + "ok": false + }, + { + "task_id": 109, + "ok": false + }, + { + "task_id": 110, + "ok": false + }, + { + "task_id": 111, + "ok": false + }, + { + "task_id": 112, + "ok": false + }, + { + "task_id": 113, + "ok": true + }, + { + "task_id": 114, + "ok": false + }, + { + "task_id": 115, + "ok": true + }, + { + "task_id": 116, + "ok": true + }, + { + "task_id": 117, + "ok": false + }, + { + "task_id": 118, + "ok": true + }, + { + "task_id": 119, + "ok": false + }, + { + "task_id": 120, + "ok": true + }, + { + "task_id": 121, + "ok": false + }, + { + "task_id": 122, + "ok": false + }, + { + "task_id": 123, + "ok": false + }, + { + "task_id": 124, + "ok": false + }, + { + "task_id": 125, + "ok": false + }, + { + "task_id": 126, + "ok": false + }, + { + "task_id": 127, + "ok": true + }, + { + "task_id": 128, + "ok": true + }, + { + "task_id": 129, + "ok": false + }, + { + "task_id": 130, + "ok": true + }, + { + "task_id": 131, + "ok": true + }, + { + "task_id": 132, + "ok": true + }, + { + "task_id": 133, + "ok": true + }, + { + "task_id": 134, + "ok": false + }, + { + "task_id": 135, + "ok": true + }, + { + "task_id": 136, + "ok": false + }, + { + "task_id": 137, + "ok": false + }, + { + "task_id": 138, + "ok": false + }, + { + "task_id": 139, + "ok": false + }, + { + "task_id": 140, + "ok": false + }, + { + "task_id": 141, + "ok": false + }, + { + "task_id": 142, + "ok": false + }, + { + "task_id": 143, + "ok": false + }, + { + "task_id": 144, + "ok": true + }, + { + "task_id": 145, + "ok": true + }, + { + "task_id": 146, + "ok": false + }, + { + "task_id": 147, + "ok": false + }, + { + "task_id": 148, + "ok": false + }, + { + "task_id": 149, + "ok": true + }, + { + "task_id": 150, + "ok": false + }, + { + "task_id": 151, + "ok": true + }, + { + "task_id": 152, + "ok": false + }, + { + "task_id": 153, + "ok": false + }, + { + "task_id": 154, + "ok": true + }, + { + "task_id": 155, + "ok": false + }, + { + "task_id": 156, + "ok": true + }, + { + "task_id": 157, + "ok": true + }, + { + "task_id": 158, + "ok": false + }, + { + "task_id": 159, + "ok": true + }, + { + "task_id": 160, + "ok": false + }, + { + "task_id": 161, + "ok": true + }, + { + "task_id": 162, + "ok": true + }, + { + "task_id": 163, + "ok": true + }, + { + "task_id": 164, + "ok": false + }, + { + "task_id": 165, + "ok": true + }, + { + "task_id": 166, + "ok": true + }, + { + "task_id": 167, + "ok": true + }, + { + "task_id": 168, + "ok": true + }, + { + "task_id": 169, + "ok": false + }, + { + "task_id": 170, + "ok": true + }, + { + "task_id": 171, + "ok": true + }, + { + "task_id": 172, + "ok": true + }, + { + "task_id": 173, + "ok": true + }, + { + "task_id": 174, + "ok": true + }, + { + "task_id": 175, + "ok": false + }, + { + "task_id": 176, + "ok": true + }, + { + "task_id": 177, + "ok": false + }, + { + "task_id": 178, + "ok": true + }, + { + "task_id": 179, + "ok": false + }, + { + "task_id": 180, + "ok": false + }, + { + "task_id": 181, + "ok": false + }, + { + "task_id": 182, + "ok": false + }, + { + "task_id": 183, + "ok": false + }, + { + "task_id": 184, + "ok": false + }, + { + "task_id": 185, + "ok": false + }, + { + "task_id": 186, + "ok": true + }, + { + "task_id": 187, + "ok": true + }, + { + "task_id": 188, + "ok": false + }, + { + "task_id": 189, + "ok": false + }, + { + "task_id": 190, + "ok": false + }, + { + "task_id": 191, + "ok": true + }, + { + "task_id": 192, + "ok": true + }, + { + "task_id": 193, + "ok": true + }, + { + "task_id": 194, + "ok": false + }, + { + "task_id": 195, + "ok": true + }, + { + "task_id": 196, + "ok": true + }, + { + "task_id": 197, + "ok": true + }, + { + "task_id": 198, + "ok": false + }, + { + "task_id": 199, + "ok": true + }, + { + "task_id": 200, + "ok": true + }, + { + "task_id": 201, + "ok": true + }, + { + "task_id": 202, + "ok": true + }, + { + "task_id": 203, + "ok": true + }, + { + "task_id": 204, + "ok": true + }, + { + "task_id": 205, + "ok": false + }, + { + "task_id": 206, + "ok": false + }, + { + "task_id": 207, + "ok": false + }, + { + "task_id": 208, + "ok": true + }, + { + "task_id": 209, + "ok": false + }, + { + "task_id": 210, + "ok": true + }, + { + "task_id": 211, + "ok": false + }, + { + "task_id": 212, + "ok": true + }, + { + "task_id": 213, + "ok": false + }, + { + "task_id": 214, + "ok": true + }, + { + "task_id": 215, + "ok": false + }, + { + "task_id": 216, + "ok": false + }, + { + "task_id": 217, + "ok": false + }, + { + "task_id": 218, + "ok": false + }, + { + "task_id": 219, + "ok": false + }, + { + "task_id": 220, + "ok": false + }, + { + "task_id": 221, + "ok": true + }, + { + "task_id": 222, + "ok": true + }, + { + "task_id": 223, + "ok": false + }, + { + "task_id": 224, + "ok": true + }, + { + "task_id": 225, + "ok": true + }, + { + "task_id": 226, + "ok": true + }, + { + "task_id": 227, + "ok": true + }, + { + "task_id": 228, + "ok": false + }, + { + "task_id": 229, + "ok": false + }, + { + "task_id": 230, + "ok": true + }, + { + "task_id": 231, + "ok": false + }, + { + "task_id": 232, + "ok": true + }, + { + "task_id": 233, + "ok": false + }, + { + "task_id": 234, + "ok": true + }, + { + "task_id": 235, + "ok": false + }, + { + "task_id": 236, + "ok": false + }, + { + "task_id": 237, + "ok": false + }, + { + "task_id": 238, + "ok": true + }, + { + "task_id": 239, + "ok": false + }, + { + "task_id": 240, + "ok": false + }, + { + "task_id": 241, + "ok": false + }, + { + "task_id": 242, + "ok": true + }, + { + "task_id": 243, + "ok": false + }, + { + "task_id": 244, + "ok": true + }, + { + "task_id": 245, + "ok": false + }, + { + "task_id": 246, + "ok": false + }, + { + "task_id": 247, + "ok": false + }, + { + "task_id": 248, + "ok": false + }, + { + "task_id": 249, + "ok": true + }, + { + "task_id": 250, + "ok": true + }, + { + "task_id": 251, + "ok": true + }, + { + "task_id": 252, + "ok": true + }, + { + "task_id": 253, + "ok": true + }, + { + "task_id": 254, + "ok": false + }, + { + "task_id": 255, + "ok": false + }, + { + "task_id": 256, + "ok": true + }, + { + "task_id": 257, + "ok": true + }, + { + "task_id": 258, + "ok": true + }, + { + "task_id": 259, + "ok": false + }, + { + "task_id": 260, + "ok": false + } + ] + } + }, + "n": 250 +} \ No newline at end of file diff --git a/results-loop/eval_lcb_distill.json b/results-loop/eval_lcb_distill.json new file mode 100644 index 0000000..bbc7ff7 --- /dev/null +++ b/results-loop/eval_lcb_distill.json @@ -0,0 +1,1221 @@ +{ + "tag": "distill", + "since": "2024-07-01", + "n": 150, + "ks": { + "0": { + "acc": 0.18666666666666668, + "hard_n": 25, + "hard_acc": 0.0, + "per_item": [ + { + "task_id": "abc382_a", + "ok": true + }, + { + "task_id": "abc382_b", + "ok": true + }, + { + "task_id": "abc382_c", + "ok": true + }, + { + "task_id": "abc382_d", + "ok": false + }, + { + "task_id": "abc382_f", + "ok": false + }, + { + "task_id": "abc382_g", + "ok": false + }, + { + "task_id": "abc383_a", + "ok": true + }, + { + "task_id": "abc383_b", + "ok": true + }, + { + "task_id": "abc383_c", + "ok": false + }, + { + "task_id": "abc383_d", + "ok": false + }, + { + "task_id": "abc383_e", + "ok": false + }, + { + "task_id": "abc384_a", + "ok": true + }, + { + "task_id": "abc384_b", + "ok": true + }, + { + "task_id": "abc384_c", + "ok": false + }, + { + "task_id": "abc384_d", + "ok": false + }, + { + "task_id": "abc384_e", + "ok": false + }, + { + "task_id": "abc384_f", + "ok": false + }, + { + "task_id": "abc384_g", + "ok": true + }, + { + "task_id": "abc385_a", + "ok": false + }, + { + "task_id": "abc385_b", + "ok": false + }, + { + "task_id": "abc385_c", + "ok": false + }, + { + "task_id": "abc385_d", + "ok": false + }, + { + "task_id": "abc385_e", + "ok": false + }, + { + "task_id": "abc385_f", + "ok": false + }, + { + "task_id": "abc386_a", + "ok": false + }, + { + "task_id": "abc386_b", + "ok": false + }, + { + "task_id": "abc386_c", + "ok": false + }, + { + "task_id": "abc386_d", + "ok": false + }, + { + "task_id": "abc386_e", + "ok": false + }, + { + "task_id": "abc386_f", + "ok": false + }, + { + "task_id": "abc387_a", + "ok": true + }, + { + "task_id": "abc387_b", + "ok": true + }, + { + "task_id": "abc387_c", + "ok": false + }, + { + "task_id": "abc387_f", + "ok": false + }, + { + "task_id": "abc388_a", + "ok": true + }, + { + "task_id": "abc388_b", + "ok": true + }, + { + "task_id": "abc388_c", + "ok": false + }, + { + "task_id": "abc388_d", + "ok": false + }, + { + "task_id": "abc388_e", + "ok": false + }, + { + "task_id": "abc388_f", + "ok": false + }, + { + "task_id": "abc388_g", + "ok": false + }, + { + "task_id": "abc389_a", + "ok": true + }, + { + "task_id": "abc389_b", + "ok": false + }, + { + "task_id": "abc389_d", + "ok": false + }, + { + "task_id": "abc389_e", + "ok": false + }, + { + "task_id": "abc389_f", + "ok": false + }, + { + "task_id": "abc389_g", + "ok": false + }, + { + "task_id": "abc390_a", + "ok": true + }, + { + "task_id": "abc390_b", + "ok": false + }, + { + "task_id": "abc390_c", + "ok": false + }, + { + "task_id": "abc390_d", + "ok": false + }, + { + "task_id": "abc390_e", + "ok": false + }, + { + "task_id": "abc390_f", + "ok": false + }, + { + "task_id": "abc390_g", + "ok": false + }, + { + "task_id": "abc391_a", + "ok": true + }, + { + "task_id": "abc391_b", + "ok": true + }, + { + "task_id": "abc391_d", + "ok": false + }, + { + "task_id": "abc391_e", + "ok": false + }, + { + "task_id": "abc391_f", + "ok": false + }, + { + "task_id": "abc391_g", + "ok": false + }, + { + "task_id": "abc392_a", + "ok": true + }, + { + "task_id": "abc392_b", + "ok": true + }, + { + "task_id": "abc392_c", + "ok": false + }, + { + "task_id": "abc392_d", + "ok": false + }, + { + "task_id": "abc392_f", + "ok": false + }, + { + "task_id": "abc392_g", + "ok": false + }, + { + "task_id": "abc393_a", + "ok": false + }, + { + "task_id": "abc393_b", + "ok": true + }, + { + "task_id": "abc393_d", + "ok": false + }, + { + "task_id": "abc393_e", + "ok": false + }, + { + "task_id": "abc393_f", + "ok": false + }, + { + "task_id": "abc394_a", + "ok": true + }, + { + "task_id": "abc394_b", + "ok": true + }, + { + "task_id": "abc394_c", + "ok": false + }, + { + "task_id": "abc394_d", + "ok": false + }, + { + "task_id": "abc394_e", + "ok": false + }, + { + "task_id": "abc394_f", + "ok": false + }, + { + "task_id": "abc394_g", + "ok": false + }, + { + "task_id": "abc395_a", + "ok": true + }, + { + "task_id": "abc395_b", + "ok": false + }, + { + "task_id": "abc395_c", + "ok": false + }, + { + "task_id": "abc395_e", + "ok": false + }, + { + "task_id": "abc395_f", + "ok": false + }, + { + "task_id": "abc396_a", + "ok": true + }, + { + "task_id": "abc396_b", + "ok": true + }, + { + "task_id": "abc396_c", + "ok": false + }, + { + "task_id": "abc396_d", + "ok": false + }, + { + "task_id": "abc396_e", + "ok": false + }, + { + "task_id": "abc396_f", + "ok": false + }, + { + "task_id": "abc396_g", + "ok": false + }, + { + "task_id": "abc397_a", + "ok": true + }, + { + "task_id": "abc397_b", + "ok": false + }, + { + "task_id": "abc397_c", + "ok": false + }, + { + "task_id": "abc397_d", + "ok": false + }, + { + "task_id": "abc397_e", + "ok": false + }, + { + "task_id": "abc397_f", + "ok": false + }, + { + "task_id": "abc397_g", + "ok": false + }, + { + "task_id": "abc398_a", + "ok": false + }, + { + "task_id": "abc398_b", + "ok": true + }, + { + "task_id": "abc398_c", + "ok": false + }, + { + "task_id": "abc398_d", + "ok": false + }, + { + "task_id": "abc398_f", + "ok": false + }, + { + "task_id": "abc398_g", + "ok": false + }, + { + "task_id": "abc399_a", + "ok": true + }, + { + "task_id": "abc399_b", + "ok": false + }, + { + "task_id": "abc399_c", + "ok": false + }, + { + "task_id": "abc399_d", + "ok": false + }, + { + "task_id": "abc399_e", + "ok": false + }, + { + "task_id": "abc399_f", + "ok": false + }, + { + "task_id": "abc400_a", + "ok": true + }, + { + "task_id": "abc400_b", + "ok": false + }, + { + "task_id": "abc400_c", + "ok": false + }, + { + "task_id": "abc400_d", + "ok": false + }, + { + "task_id": "abc400_e", + "ok": false + }, + { + "task_id": "abc400_g", + "ok": false + }, + { + "task_id": "arc188_a", + "ok": false + }, + { + "task_id": "arc188_b", + "ok": false + }, + { + "task_id": "arc188_c", + "ok": false + }, + { + "task_id": "arc188_d", + "ok": false + }, + { + "task_id": "arc189_a", + "ok": false + }, + { + "task_id": "arc189_b", + "ok": false + }, + { + "task_id": "arc189_c", + "ok": false + }, + { + "task_id": "arc189_d", + "ok": false + }, + { + "task_id": "arc190_a", + "ok": false + }, + { + "task_id": "arc190_c", + "ok": false + }, + { + "task_id": "arc190_d", + "ok": false + }, + { + "task_id": "arc191_a", + "ok": false + }, + { + "task_id": "arc191_c", + "ok": false + }, + { + "task_id": "arc191_d", + "ok": false + }, + { + "task_id": "arc192_a", + "ok": false + }, + { + "task_id": "arc192_b", + "ok": false + }, + { + "task_id": "arc192_d", + "ok": false + }, + { + "task_id": "arc192_e", + "ok": false + }, + { + "task_id": "arc193_a", + "ok": false + }, + { + "task_id": "arc193_b", + "ok": false + }, + { + "task_id": "arc193_d", + "ok": false + }, + { + "task_id": "arc194_a", + "ok": false + }, + { + "task_id": "arc194_b", + "ok": false + }, + { + "task_id": "arc194_c", + "ok": false + }, + { + "task_id": "arc194_d", + "ok": false + }, + { + "task_id": "arc194_e", + "ok": false + }, + { + "task_id": "arc195_a", + "ok": false + }, + { + "task_id": "arc195_b", + "ok": false + }, + { + "task_id": "arc195_c", + "ok": false + }, + { + "task_id": "arc195_d", + "ok": false + }, + { + "task_id": "arc195_e", + "ok": false + }, + { + "task_id": "arc196_a", + "ok": false + }, + { + "task_id": "arc196_b", + "ok": false + }, + { + "task_id": "arc196_c", + "ok": false + }, + { + "task_id": "arc196_d", + "ok": false + } + ] + }, + "1": { + "acc": 0.12666666666666668, + "hard_n": 25, + "hard_acc": 0.16, + "per_item": [ + { + "task_id": "abc382_a", + "ok": true + }, + { + "task_id": "abc382_b", + "ok": false + }, + { + "task_id": "abc382_c", + "ok": false + }, + { + "task_id": "abc382_d", + "ok": true + }, + { + "task_id": "abc382_f", + "ok": false + }, + { + "task_id": "abc382_g", + "ok": false + }, + { + "task_id": "abc383_a", + "ok": false + }, + { + "task_id": "abc383_b", + "ok": false + }, + { + "task_id": "abc383_c", + "ok": false + }, + { + "task_id": "abc383_d", + "ok": false + }, + { + "task_id": "abc383_e", + "ok": false + }, + { + "task_id": "abc384_a", + "ok": true + }, + { + "task_id": "abc384_b", + "ok": true + }, + { + "task_id": "abc384_c", + "ok": false + }, + { + "task_id": "abc384_d", + "ok": false + }, + { + "task_id": "abc384_e", + "ok": false + }, + { + "task_id": "abc384_f", + "ok": false + }, + { + "task_id": "abc384_g", + "ok": false + }, + { + "task_id": "abc385_a", + "ok": false + }, + { + "task_id": "abc385_b", + "ok": true + }, + { + "task_id": "abc385_c", + "ok": false + }, + { + "task_id": "abc385_d", + "ok": false + }, + { + "task_id": "abc385_e", + "ok": false + }, + { + "task_id": "abc385_f", + "ok": false + }, + { + "task_id": "abc386_a", + "ok": false + }, + { + "task_id": "abc386_b", + "ok": false + }, + { + "task_id": "abc386_c", + "ok": false + }, + { + "task_id": "abc386_d", + "ok": false + }, + { + "task_id": "abc386_e", + "ok": false + }, + { + "task_id": "abc386_f", + "ok": false + }, + { + "task_id": "abc387_a", + "ok": true + }, + { + "task_id": "abc387_b", + "ok": false + }, + { + "task_id": "abc387_c", + "ok": false + }, + { + "task_id": "abc387_f", + "ok": false + }, + { + "task_id": "abc388_a", + "ok": true + }, + { + "task_id": "abc388_b", + "ok": false + }, + { + "task_id": "abc388_c", + "ok": false + }, + { + "task_id": "abc388_d", + "ok": false + }, + { + "task_id": "abc388_e", + "ok": false + }, + { + "task_id": "abc388_f", + "ok": false + }, + { + "task_id": "abc388_g", + "ok": false + }, + { + "task_id": "abc389_a", + "ok": true + }, + { + "task_id": "abc389_b", + "ok": false + }, + { + "task_id": "abc389_d", + "ok": false + }, + { + "task_id": "abc389_e", + "ok": false + }, + { + "task_id": "abc389_f", + "ok": false + }, + { + "task_id": "abc389_g", + "ok": false + }, + { + "task_id": "abc390_a", + "ok": false + }, + { + "task_id": "abc390_b", + "ok": false + }, + { + "task_id": "abc390_c", + "ok": false + }, + { + "task_id": "abc390_d", + "ok": false + }, + { + "task_id": "abc390_e", + "ok": false + }, + { + "task_id": "abc390_f", + "ok": false + }, + { + "task_id": "abc390_g", + "ok": false + }, + { + "task_id": "abc391_a", + "ok": true + }, + { + "task_id": "abc391_b", + "ok": false + }, + { + "task_id": "abc391_d", + "ok": false + }, + { + "task_id": "abc391_e", + "ok": false + }, + { + "task_id": "abc391_f", + "ok": false + }, + { + "task_id": "abc391_g", + "ok": false + }, + { + "task_id": "abc392_a", + "ok": false + }, + { + "task_id": "abc392_b", + "ok": true + }, + { + "task_id": "abc392_c", + "ok": false + }, + { + "task_id": "abc392_d", + "ok": false + }, + { + "task_id": "abc392_f", + "ok": false + }, + { + "task_id": "abc392_g", + "ok": false + }, + { + "task_id": "abc393_a", + "ok": false + }, + { + "task_id": "abc393_b", + "ok": false + }, + { + "task_id": "abc393_d", + "ok": false + }, + { + "task_id": "abc393_e", + "ok": false + }, + { + "task_id": "abc393_f", + "ok": false + }, + { + "task_id": "abc394_a", + "ok": true + }, + { + "task_id": "abc394_b", + "ok": true + }, + { + "task_id": "abc394_c", + "ok": true + }, + { + "task_id": "abc394_d", + "ok": false + }, + { + "task_id": "abc394_e", + "ok": false + }, + { + "task_id": "abc394_f", + "ok": false + }, + { + "task_id": "abc394_g", + "ok": false + }, + { + "task_id": "abc395_a", + "ok": true + }, + { + "task_id": "abc395_b", + "ok": true + }, + { + "task_id": "abc395_c", + "ok": false + }, + { + "task_id": "abc395_e", + "ok": false + }, + { + "task_id": "abc395_f", + "ok": false + }, + { + "task_id": "abc396_a", + "ok": true + }, + { + "task_id": "abc396_b", + "ok": true + }, + { + "task_id": "abc396_c", + "ok": false + }, + { + "task_id": "abc396_d", + "ok": false + }, + { + "task_id": "abc396_e", + "ok": false + }, + { + "task_id": "abc396_f", + "ok": false + }, + { + "task_id": "abc396_g", + "ok": false + }, + { + "task_id": "abc397_a", + "ok": true + }, + { + "task_id": "abc397_b", + "ok": false + }, + { + "task_id": "abc397_c", + "ok": false + }, + { + "task_id": "abc397_d", + "ok": false + }, + { + "task_id": "abc397_e", + "ok": false + }, + { + "task_id": "abc397_f", + "ok": false + }, + { + "task_id": "abc397_g", + "ok": false + }, + { + "task_id": "abc398_a", + "ok": false + }, + { + "task_id": "abc398_b", + "ok": false + }, + { + "task_id": "abc398_c", + "ok": false + }, + { + "task_id": "abc398_d", + "ok": false + }, + { + "task_id": "abc398_f", + "ok": false + }, + { + "task_id": "abc398_g", + "ok": false + }, + { + "task_id": "abc399_a", + "ok": true + }, + { + "task_id": "abc399_b", + "ok": false + }, + { + "task_id": "abc399_c", + "ok": false + }, + { + "task_id": "abc399_d", + "ok": false + }, + { + "task_id": "abc399_e", + "ok": false + }, + { + "task_id": "abc399_f", + "ok": false + }, + { + "task_id": "abc400_a", + "ok": false + }, + { + "task_id": "abc400_b", + "ok": false + }, + { + "task_id": "abc400_c", + "ok": false + }, + { + "task_id": "abc400_d", + "ok": false + }, + { + "task_id": "abc400_e", + "ok": false + }, + { + "task_id": "abc400_g", + "ok": false + }, + { + "task_id": "arc188_a", + "ok": false + }, + { + "task_id": "arc188_b", + "ok": false + }, + { + "task_id": "arc188_c", + "ok": false + }, + { + "task_id": "arc188_d", + "ok": false + }, + { + "task_id": "arc189_a", + "ok": false + }, + { + "task_id": "arc189_b", + "ok": false + }, + { + "task_id": "arc189_c", + "ok": false + }, + { + "task_id": "arc189_d", + "ok": false + }, + { + "task_id": "arc190_a", + "ok": false + }, + { + "task_id": "arc190_c", + "ok": false + }, + { + "task_id": "arc190_d", + "ok": false + }, + { + "task_id": "arc191_a", + "ok": false + }, + { + "task_id": "arc191_c", + "ok": false + }, + { + "task_id": "arc191_d", + "ok": false + }, + { + "task_id": "arc192_a", + "ok": false + }, + { + "task_id": "arc192_b", + "ok": false + }, + { + "task_id": "arc192_d", + "ok": false + }, + { + "task_id": "arc192_e", + "ok": false + }, + { + "task_id": "arc193_a", + "ok": false + }, + { + "task_id": "arc193_b", + "ok": false + }, + { + "task_id": "arc193_d", + "ok": false + }, + { + "task_id": "arc194_a", + "ok": false + }, + { + "task_id": "arc194_b", + "ok": false + }, + { + "task_id": "arc194_c", + "ok": false + }, + { + "task_id": "arc194_d", + "ok": false + }, + { + "task_id": "arc194_e", + "ok": false + }, + { + "task_id": "arc195_a", + "ok": false + }, + { + "task_id": "arc195_b", + "ok": false + }, + { + "task_id": "arc195_c", + "ok": false + }, + { + "task_id": "arc195_d", + "ok": false + }, + { + "task_id": "arc195_e", + "ok": false + }, + { + "task_id": "arc196_a", + "ok": false + }, + { + "task_id": "arc196_b", + "ok": false + }, + { + "task_id": "arc196_c", + "ok": false + }, + { + "task_id": "arc196_d", + "ok": false + } + ] + } + } +} \ No newline at end of file diff --git a/results-loop/eval_lcb_loop.json b/results-loop/eval_lcb_loop.json new file mode 100644 index 0000000..a3397e2 --- /dev/null +++ b/results-loop/eval_lcb_loop.json @@ -0,0 +1,1221 @@ +{ + "tag": "loop", + "since": "2024-07-01", + "n": 150, + "ks": { + "0": { + "acc": 0.18666666666666668, + "hard_n": 25, + "hard_acc": 0.0, + "per_item": [ + { + "task_id": "abc382_a", + "ok": true + }, + { + "task_id": "abc382_b", + "ok": true + }, + { + "task_id": "abc382_c", + "ok": true + }, + { + "task_id": "abc382_d", + "ok": false + }, + { + "task_id": "abc382_f", + "ok": false + }, + { + "task_id": "abc382_g", + "ok": false + }, + { + "task_id": "abc383_a", + "ok": true + }, + { + "task_id": "abc383_b", + "ok": true + }, + { + "task_id": "abc383_c", + "ok": false + }, + { + "task_id": "abc383_d", + "ok": false + }, + { + "task_id": "abc383_e", + "ok": false + }, + { + "task_id": "abc384_a", + "ok": true + }, + { + "task_id": "abc384_b", + "ok": true + }, + { + "task_id": "abc384_c", + "ok": false + }, + { + "task_id": "abc384_d", + "ok": false + }, + { + "task_id": "abc384_e", + "ok": false + }, + { + "task_id": "abc384_f", + "ok": false + }, + { + "task_id": "abc384_g", + "ok": true + }, + { + "task_id": "abc385_a", + "ok": false + }, + { + "task_id": "abc385_b", + "ok": false + }, + { + "task_id": "abc385_c", + "ok": false + }, + { + "task_id": "abc385_d", + "ok": false + }, + { + "task_id": "abc385_e", + "ok": false + }, + { + "task_id": "abc385_f", + "ok": false + }, + { + "task_id": "abc386_a", + "ok": false + }, + { + "task_id": "abc386_b", + "ok": false + }, + { + "task_id": "abc386_c", + "ok": false + }, + { + "task_id": "abc386_d", + "ok": false + }, + { + "task_id": "abc386_e", + "ok": false + }, + { + "task_id": "abc386_f", + "ok": false + }, + { + "task_id": "abc387_a", + "ok": true + }, + { + "task_id": "abc387_b", + "ok": true + }, + { + "task_id": "abc387_c", + "ok": false + }, + { + "task_id": "abc387_f", + "ok": false + }, + { + "task_id": "abc388_a", + "ok": true + }, + { + "task_id": "abc388_b", + "ok": true + }, + { + "task_id": "abc388_c", + "ok": false + }, + { + "task_id": "abc388_d", + "ok": false + }, + { + "task_id": "abc388_e", + "ok": false + }, + { + "task_id": "abc388_f", + "ok": false + }, + { + "task_id": "abc388_g", + "ok": false + }, + { + "task_id": "abc389_a", + "ok": true + }, + { + "task_id": "abc389_b", + "ok": false + }, + { + "task_id": "abc389_d", + "ok": false + }, + { + "task_id": "abc389_e", + "ok": false + }, + { + "task_id": "abc389_f", + "ok": false + }, + { + "task_id": "abc389_g", + "ok": false + }, + { + "task_id": "abc390_a", + "ok": true + }, + { + "task_id": "abc390_b", + "ok": false + }, + { + "task_id": "abc390_c", + "ok": false + }, + { + "task_id": "abc390_d", + "ok": false + }, + { + "task_id": "abc390_e", + "ok": false + }, + { + "task_id": "abc390_f", + "ok": false + }, + { + "task_id": "abc390_g", + "ok": false + }, + { + "task_id": "abc391_a", + "ok": true + }, + { + "task_id": "abc391_b", + "ok": true + }, + { + "task_id": "abc391_d", + "ok": false + }, + { + "task_id": "abc391_e", + "ok": false + }, + { + "task_id": "abc391_f", + "ok": false + }, + { + "task_id": "abc391_g", + "ok": false + }, + { + "task_id": "abc392_a", + "ok": true + }, + { + "task_id": "abc392_b", + "ok": true + }, + { + "task_id": "abc392_c", + "ok": false + }, + { + "task_id": "abc392_d", + "ok": false + }, + { + "task_id": "abc392_f", + "ok": false + }, + { + "task_id": "abc392_g", + "ok": false + }, + { + "task_id": "abc393_a", + "ok": false + }, + { + "task_id": "abc393_b", + "ok": true + }, + { + "task_id": "abc393_d", + "ok": false + }, + { + "task_id": "abc393_e", + "ok": false + }, + { + "task_id": "abc393_f", + "ok": false + }, + { + "task_id": "abc394_a", + "ok": true + }, + { + "task_id": "abc394_b", + "ok": true + }, + { + "task_id": "abc394_c", + "ok": false + }, + { + "task_id": "abc394_d", + "ok": false + }, + { + "task_id": "abc394_e", + "ok": false + }, + { + "task_id": "abc394_f", + "ok": false + }, + { + "task_id": "abc394_g", + "ok": false + }, + { + "task_id": "abc395_a", + "ok": true + }, + { + "task_id": "abc395_b", + "ok": false + }, + { + "task_id": "abc395_c", + "ok": false + }, + { + "task_id": "abc395_e", + "ok": false + }, + { + "task_id": "abc395_f", + "ok": false + }, + { + "task_id": "abc396_a", + "ok": true + }, + { + "task_id": "abc396_b", + "ok": true + }, + { + "task_id": "abc396_c", + "ok": false + }, + { + "task_id": "abc396_d", + "ok": false + }, + { + "task_id": "abc396_e", + "ok": false + }, + { + "task_id": "abc396_f", + "ok": false + }, + { + "task_id": "abc396_g", + "ok": false + }, + { + "task_id": "abc397_a", + "ok": true + }, + { + "task_id": "abc397_b", + "ok": false + }, + { + "task_id": "abc397_c", + "ok": false + }, + { + "task_id": "abc397_d", + "ok": false + }, + { + "task_id": "abc397_e", + "ok": false + }, + { + "task_id": "abc397_f", + "ok": false + }, + { + "task_id": "abc397_g", + "ok": false + }, + { + "task_id": "abc398_a", + "ok": false + }, + { + "task_id": "abc398_b", + "ok": true + }, + { + "task_id": "abc398_c", + "ok": false + }, + { + "task_id": "abc398_d", + "ok": false + }, + { + "task_id": "abc398_f", + "ok": false + }, + { + "task_id": "abc398_g", + "ok": false + }, + { + "task_id": "abc399_a", + "ok": true + }, + { + "task_id": "abc399_b", + "ok": false + }, + { + "task_id": "abc399_c", + "ok": false + }, + { + "task_id": "abc399_d", + "ok": false + }, + { + "task_id": "abc399_e", + "ok": false + }, + { + "task_id": "abc399_f", + "ok": false + }, + { + "task_id": "abc400_a", + "ok": true + }, + { + "task_id": "abc400_b", + "ok": false + }, + { + "task_id": "abc400_c", + "ok": false + }, + { + "task_id": "abc400_d", + "ok": false + }, + { + "task_id": "abc400_e", + "ok": false + }, + { + "task_id": "abc400_g", + "ok": false + }, + { + "task_id": "arc188_a", + "ok": false + }, + { + "task_id": "arc188_b", + "ok": false + }, + { + "task_id": "arc188_c", + "ok": false + }, + { + "task_id": "arc188_d", + "ok": false + }, + { + "task_id": "arc189_a", + "ok": false + }, + { + "task_id": "arc189_b", + "ok": false + }, + { + "task_id": "arc189_c", + "ok": false + }, + { + "task_id": "arc189_d", + "ok": false + }, + { + "task_id": "arc190_a", + "ok": false + }, + { + "task_id": "arc190_c", + "ok": false + }, + { + "task_id": "arc190_d", + "ok": false + }, + { + "task_id": "arc191_a", + "ok": false + }, + { + "task_id": "arc191_c", + "ok": false + }, + { + "task_id": "arc191_d", + "ok": false + }, + { + "task_id": "arc192_a", + "ok": false + }, + { + "task_id": "arc192_b", + "ok": false + }, + { + "task_id": "arc192_d", + "ok": false + }, + { + "task_id": "arc192_e", + "ok": false + }, + { + "task_id": "arc193_a", + "ok": false + }, + { + "task_id": "arc193_b", + "ok": false + }, + { + "task_id": "arc193_d", + "ok": false + }, + { + "task_id": "arc194_a", + "ok": false + }, + { + "task_id": "arc194_b", + "ok": false + }, + { + "task_id": "arc194_c", + "ok": false + }, + { + "task_id": "arc194_d", + "ok": false + }, + { + "task_id": "arc194_e", + "ok": false + }, + { + "task_id": "arc195_a", + "ok": false + }, + { + "task_id": "arc195_b", + "ok": false + }, + { + "task_id": "arc195_c", + "ok": false + }, + { + "task_id": "arc195_d", + "ok": false + }, + { + "task_id": "arc195_e", + "ok": false + }, + { + "task_id": "arc196_a", + "ok": false + }, + { + "task_id": "arc196_b", + "ok": false + }, + { + "task_id": "arc196_c", + "ok": false + }, + { + "task_id": "arc196_d", + "ok": false + } + ] + }, + "4": { + "acc": 0.15333333333333332, + "hard_n": 25, + "hard_acc": 0.08, + "per_item": [ + { + "task_id": "abc382_a", + "ok": true + }, + { + "task_id": "abc382_b", + "ok": false + }, + { + "task_id": "abc382_c", + "ok": false + }, + { + "task_id": "abc382_d", + "ok": false + }, + { + "task_id": "abc382_f", + "ok": false + }, + { + "task_id": "abc382_g", + "ok": false + }, + { + "task_id": "abc383_a", + "ok": false + }, + { + "task_id": "abc383_b", + "ok": false + }, + { + "task_id": "abc383_c", + "ok": false + }, + { + "task_id": "abc383_d", + "ok": false + }, + { + "task_id": "abc383_e", + "ok": false + }, + { + "task_id": "abc384_a", + "ok": true + }, + { + "task_id": "abc384_b", + "ok": true + }, + { + "task_id": "abc384_c", + "ok": false + }, + { + "task_id": "abc384_d", + "ok": false + }, + { + "task_id": "abc384_e", + "ok": false + }, + { + "task_id": "abc384_f", + "ok": false + }, + { + "task_id": "abc384_g", + "ok": false + }, + { + "task_id": "abc385_a", + "ok": false + }, + { + "task_id": "abc385_b", + "ok": false + }, + { + "task_id": "abc385_c", + "ok": true + }, + { + "task_id": "abc385_d", + "ok": false + }, + { + "task_id": "abc385_e", + "ok": false + }, + { + "task_id": "abc385_f", + "ok": false + }, + { + "task_id": "abc386_a", + "ok": false + }, + { + "task_id": "abc386_b", + "ok": false + }, + { + "task_id": "abc386_c", + "ok": false + }, + { + "task_id": "abc386_d", + "ok": false + }, + { + "task_id": "abc386_e", + "ok": false + }, + { + "task_id": "abc386_f", + "ok": false + }, + { + "task_id": "abc387_a", + "ok": true + }, + { + "task_id": "abc387_b", + "ok": true + }, + { + "task_id": "abc387_c", + "ok": false + }, + { + "task_id": "abc387_f", + "ok": false + }, + { + "task_id": "abc388_a", + "ok": true + }, + { + "task_id": "abc388_b", + "ok": false + }, + { + "task_id": "abc388_c", + "ok": false + }, + { + "task_id": "abc388_d", + "ok": false + }, + { + "task_id": "abc388_e", + "ok": false + }, + { + "task_id": "abc388_f", + "ok": false + }, + { + "task_id": "abc388_g", + "ok": false + }, + { + "task_id": "abc389_a", + "ok": true + }, + { + "task_id": "abc389_b", + "ok": true + }, + { + "task_id": "abc389_d", + "ok": false + }, + { + "task_id": "abc389_e", + "ok": false + }, + { + "task_id": "abc389_f", + "ok": false + }, + { + "task_id": "abc389_g", + "ok": false + }, + { + "task_id": "abc390_a", + "ok": true + }, + { + "task_id": "abc390_b", + "ok": false + }, + { + "task_id": "abc390_c", + "ok": false + }, + { + "task_id": "abc390_d", + "ok": false + }, + { + "task_id": "abc390_e", + "ok": false + }, + { + "task_id": "abc390_f", + "ok": false + }, + { + "task_id": "abc390_g", + "ok": false + }, + { + "task_id": "abc391_a", + "ok": true + }, + { + "task_id": "abc391_b", + "ok": false + }, + { + "task_id": "abc391_d", + "ok": false + }, + { + "task_id": "abc391_e", + "ok": false + }, + { + "task_id": "abc391_f", + "ok": false + }, + { + "task_id": "abc391_g", + "ok": false + }, + { + "task_id": "abc392_a", + "ok": true + }, + { + "task_id": "abc392_b", + "ok": true + }, + { + "task_id": "abc392_c", + "ok": false + }, + { + "task_id": "abc392_d", + "ok": false + }, + { + "task_id": "abc392_f", + "ok": false + }, + { + "task_id": "abc392_g", + "ok": false + }, + { + "task_id": "abc393_a", + "ok": true + }, + { + "task_id": "abc393_b", + "ok": false + }, + { + "task_id": "abc393_d", + "ok": false + }, + { + "task_id": "abc393_e", + "ok": false + }, + { + "task_id": "abc393_f", + "ok": false + }, + { + "task_id": "abc394_a", + "ok": true + }, + { + "task_id": "abc394_b", + "ok": true + }, + { + "task_id": "abc394_c", + "ok": false + }, + { + "task_id": "abc394_d", + "ok": false + }, + { + "task_id": "abc394_e", + "ok": false + }, + { + "task_id": "abc394_f", + "ok": false + }, + { + "task_id": "abc394_g", + "ok": false + }, + { + "task_id": "abc395_a", + "ok": true + }, + { + "task_id": "abc395_b", + "ok": false + }, + { + "task_id": "abc395_c", + "ok": false + }, + { + "task_id": "abc395_e", + "ok": false + }, + { + "task_id": "abc395_f", + "ok": false + }, + { + "task_id": "abc396_a", + "ok": true + }, + { + "task_id": "abc396_b", + "ok": true + }, + { + "task_id": "abc396_c", + "ok": false + }, + { + "task_id": "abc396_d", + "ok": false + }, + { + "task_id": "abc396_e", + "ok": false + }, + { + "task_id": "abc396_f", + "ok": false + }, + { + "task_id": "abc396_g", + "ok": false + }, + { + "task_id": "abc397_a", + "ok": true + }, + { + "task_id": "abc397_b", + "ok": false + }, + { + "task_id": "abc397_c", + "ok": false + }, + { + "task_id": "abc397_d", + "ok": false + }, + { + "task_id": "abc397_e", + "ok": false + }, + { + "task_id": "abc397_f", + "ok": false + }, + { + "task_id": "abc397_g", + "ok": false + }, + { + "task_id": "abc398_a", + "ok": false + }, + { + "task_id": "abc398_b", + "ok": true + }, + { + "task_id": "abc398_c", + "ok": false + }, + { + "task_id": "abc398_d", + "ok": false + }, + { + "task_id": "abc398_f", + "ok": false + }, + { + "task_id": "abc398_g", + "ok": false + }, + { + "task_id": "abc399_a", + "ok": true + }, + { + "task_id": "abc399_b", + "ok": false + }, + { + "task_id": "abc399_c", + "ok": false + }, + { + "task_id": "abc399_d", + "ok": false + }, + { + "task_id": "abc399_e", + "ok": false + }, + { + "task_id": "abc399_f", + "ok": false + }, + { + "task_id": "abc400_a", + "ok": true + }, + { + "task_id": "abc400_b", + "ok": false + }, + { + "task_id": "abc400_c", + "ok": false + }, + { + "task_id": "abc400_d", + "ok": false + }, + { + "task_id": "abc400_e", + "ok": false + }, + { + "task_id": "abc400_g", + "ok": false + }, + { + "task_id": "arc188_a", + "ok": false + }, + { + "task_id": "arc188_b", + "ok": false + }, + { + "task_id": "arc188_c", + "ok": false + }, + { + "task_id": "arc188_d", + "ok": false + }, + { + "task_id": "arc189_a", + "ok": false + }, + { + "task_id": "arc189_b", + "ok": false + }, + { + "task_id": "arc189_c", + "ok": false + }, + { + "task_id": "arc189_d", + "ok": false + }, + { + "task_id": "arc190_a", + "ok": false + }, + { + "task_id": "arc190_c", + "ok": false + }, + { + "task_id": "arc190_d", + "ok": false + }, + { + "task_id": "arc191_a", + "ok": false + }, + { + "task_id": "arc191_c", + "ok": false + }, + { + "task_id": "arc191_d", + "ok": false + }, + { + "task_id": "arc192_a", + "ok": false + }, + { + "task_id": "arc192_b", + "ok": false + }, + { + "task_id": "arc192_d", + "ok": false + }, + { + "task_id": "arc192_e", + "ok": false + }, + { + "task_id": "arc193_a", + "ok": false + }, + { + "task_id": "arc193_b", + "ok": false + }, + { + "task_id": "arc193_d", + "ok": false + }, + { + "task_id": "arc194_a", + "ok": false + }, + { + "task_id": "arc194_b", + "ok": false + }, + { + "task_id": "arc194_c", + "ok": false + }, + { + "task_id": "arc194_d", + "ok": false + }, + { + "task_id": "arc194_e", + "ok": false + }, + { + "task_id": "arc195_a", + "ok": false + }, + { + "task_id": "arc195_b", + "ok": false + }, + { + "task_id": "arc195_c", + "ok": false + }, + { + "task_id": "arc195_d", + "ok": false + }, + { + "task_id": "arc195_e", + "ok": false + }, + { + "task_id": "arc196_a", + "ok": false + }, + { + "task_id": "arc196_b", + "ok": false + }, + { + "task_id": "arc196_c", + "ok": false + }, + { + "task_id": "arc196_d", + "ok": false + } + ] + } + } +} \ No newline at end of file diff --git a/results-loop/eval_lcb_untrained.json b/results-loop/eval_lcb_untrained.json new file mode 100644 index 0000000..7ff2ad3 --- /dev/null +++ b/results-loop/eval_lcb_untrained.json @@ -0,0 +1,1221 @@ +{ + "tag": "untrained", + "since": "2024-07-01", + "n": 150, + "ks": { + "0": { + "acc": 0.18666666666666668, + "hard_n": 25, + "hard_acc": 0.0, + "per_item": [ + { + "task_id": "abc382_a", + "ok": true + }, + { + "task_id": "abc382_b", + "ok": true + }, + { + "task_id": "abc382_c", + "ok": true + }, + { + "task_id": "abc382_d", + "ok": false + }, + { + "task_id": "abc382_f", + "ok": false + }, + { + "task_id": "abc382_g", + "ok": false + }, + { + "task_id": "abc383_a", + "ok": true + }, + { + "task_id": "abc383_b", + "ok": true + }, + { + "task_id": "abc383_c", + "ok": false + }, + { + "task_id": "abc383_d", + "ok": false + }, + { + "task_id": "abc383_e", + "ok": false + }, + { + "task_id": "abc384_a", + "ok": true + }, + { + "task_id": "abc384_b", + "ok": true + }, + { + "task_id": "abc384_c", + "ok": false + }, + { + "task_id": "abc384_d", + "ok": false + }, + { + "task_id": "abc384_e", + "ok": false + }, + { + "task_id": "abc384_f", + "ok": false + }, + { + "task_id": "abc384_g", + "ok": true + }, + { + "task_id": "abc385_a", + "ok": false + }, + { + "task_id": "abc385_b", + "ok": false + }, + { + "task_id": "abc385_c", + "ok": false + }, + { + "task_id": "abc385_d", + "ok": false + }, + { + "task_id": "abc385_e", + "ok": false + }, + { + "task_id": "abc385_f", + "ok": false + }, + { + "task_id": "abc386_a", + "ok": false + }, + { + "task_id": "abc386_b", + "ok": false + }, + { + "task_id": "abc386_c", + "ok": false + }, + { + "task_id": "abc386_d", + "ok": false + }, + { + "task_id": "abc386_e", + "ok": false + }, + { + "task_id": "abc386_f", + "ok": false + }, + { + "task_id": "abc387_a", + "ok": true + }, + { + "task_id": "abc387_b", + "ok": true + }, + { + "task_id": "abc387_c", + "ok": false + }, + { + "task_id": "abc387_f", + "ok": false + }, + { + "task_id": "abc388_a", + "ok": true + }, + { + "task_id": "abc388_b", + "ok": true + }, + { + "task_id": "abc388_c", + "ok": false + }, + { + "task_id": "abc388_d", + "ok": false + }, + { + "task_id": "abc388_e", + "ok": false + }, + { + "task_id": "abc388_f", + "ok": false + }, + { + "task_id": "abc388_g", + "ok": false + }, + { + "task_id": "abc389_a", + "ok": true + }, + { + "task_id": "abc389_b", + "ok": false + }, + { + "task_id": "abc389_d", + "ok": false + }, + { + "task_id": "abc389_e", + "ok": false + }, + { + "task_id": "abc389_f", + "ok": false + }, + { + "task_id": "abc389_g", + "ok": false + }, + { + "task_id": "abc390_a", + "ok": true + }, + { + "task_id": "abc390_b", + "ok": false + }, + { + "task_id": "abc390_c", + "ok": false + }, + { + "task_id": "abc390_d", + "ok": false + }, + { + "task_id": "abc390_e", + "ok": false + }, + { + "task_id": "abc390_f", + "ok": false + }, + { + "task_id": "abc390_g", + "ok": false + }, + { + "task_id": "abc391_a", + "ok": true + }, + { + "task_id": "abc391_b", + "ok": true + }, + { + "task_id": "abc391_d", + "ok": false + }, + { + "task_id": "abc391_e", + "ok": false + }, + { + "task_id": "abc391_f", + "ok": false + }, + { + "task_id": "abc391_g", + "ok": false + }, + { + "task_id": "abc392_a", + "ok": true + }, + { + "task_id": "abc392_b", + "ok": true + }, + { + "task_id": "abc392_c", + "ok": false + }, + { + "task_id": "abc392_d", + "ok": false + }, + { + "task_id": "abc392_f", + "ok": false + }, + { + "task_id": "abc392_g", + "ok": false + }, + { + "task_id": "abc393_a", + "ok": false + }, + { + "task_id": "abc393_b", + "ok": true + }, + { + "task_id": "abc393_d", + "ok": false + }, + { + "task_id": "abc393_e", + "ok": false + }, + { + "task_id": "abc393_f", + "ok": false + }, + { + "task_id": "abc394_a", + "ok": true + }, + { + "task_id": "abc394_b", + "ok": true + }, + { + "task_id": "abc394_c", + "ok": false + }, + { + "task_id": "abc394_d", + "ok": false + }, + { + "task_id": "abc394_e", + "ok": false + }, + { + "task_id": "abc394_f", + "ok": false + }, + { + "task_id": "abc394_g", + "ok": false + }, + { + "task_id": "abc395_a", + "ok": true + }, + { + "task_id": "abc395_b", + "ok": false + }, + { + "task_id": "abc395_c", + "ok": false + }, + { + "task_id": "abc395_e", + "ok": false + }, + { + "task_id": "abc395_f", + "ok": false + }, + { + "task_id": "abc396_a", + "ok": true + }, + { + "task_id": "abc396_b", + "ok": true + }, + { + "task_id": "abc396_c", + "ok": false + }, + { + "task_id": "abc396_d", + "ok": false + }, + { + "task_id": "abc396_e", + "ok": false + }, + { + "task_id": "abc396_f", + "ok": false + }, + { + "task_id": "abc396_g", + "ok": false + }, + { + "task_id": "abc397_a", + "ok": true + }, + { + "task_id": "abc397_b", + "ok": false + }, + { + "task_id": "abc397_c", + "ok": false + }, + { + "task_id": "abc397_d", + "ok": false + }, + { + "task_id": "abc397_e", + "ok": false + }, + { + "task_id": "abc397_f", + "ok": false + }, + { + "task_id": "abc397_g", + "ok": false + }, + { + "task_id": "abc398_a", + "ok": false + }, + { + "task_id": "abc398_b", + "ok": true + }, + { + "task_id": "abc398_c", + "ok": false + }, + { + "task_id": "abc398_d", + "ok": false + }, + { + "task_id": "abc398_f", + "ok": false + }, + { + "task_id": "abc398_g", + "ok": false + }, + { + "task_id": "abc399_a", + "ok": true + }, + { + "task_id": "abc399_b", + "ok": false + }, + { + "task_id": "abc399_c", + "ok": false + }, + { + "task_id": "abc399_d", + "ok": false + }, + { + "task_id": "abc399_e", + "ok": false + }, + { + "task_id": "abc399_f", + "ok": false + }, + { + "task_id": "abc400_a", + "ok": true + }, + { + "task_id": "abc400_b", + "ok": false + }, + { + "task_id": "abc400_c", + "ok": false + }, + { + "task_id": "abc400_d", + "ok": false + }, + { + "task_id": "abc400_e", + "ok": false + }, + { + "task_id": "abc400_g", + "ok": false + }, + { + "task_id": "arc188_a", + "ok": false + }, + { + "task_id": "arc188_b", + "ok": false + }, + { + "task_id": "arc188_c", + "ok": false + }, + { + "task_id": "arc188_d", + "ok": false + }, + { + "task_id": "arc189_a", + "ok": false + }, + { + "task_id": "arc189_b", + "ok": false + }, + { + "task_id": "arc189_c", + "ok": false + }, + { + "task_id": "arc189_d", + "ok": false + }, + { + "task_id": "arc190_a", + "ok": false + }, + { + "task_id": "arc190_c", + "ok": false + }, + { + "task_id": "arc190_d", + "ok": false + }, + { + "task_id": "arc191_a", + "ok": false + }, + { + "task_id": "arc191_c", + "ok": false + }, + { + "task_id": "arc191_d", + "ok": false + }, + { + "task_id": "arc192_a", + "ok": false + }, + { + "task_id": "arc192_b", + "ok": false + }, + { + "task_id": "arc192_d", + "ok": false + }, + { + "task_id": "arc192_e", + "ok": false + }, + { + "task_id": "arc193_a", + "ok": false + }, + { + "task_id": "arc193_b", + "ok": false + }, + { + "task_id": "arc193_d", + "ok": false + }, + { + "task_id": "arc194_a", + "ok": false + }, + { + "task_id": "arc194_b", + "ok": false + }, + { + "task_id": "arc194_c", + "ok": false + }, + { + "task_id": "arc194_d", + "ok": false + }, + { + "task_id": "arc194_e", + "ok": false + }, + { + "task_id": "arc195_a", + "ok": false + }, + { + "task_id": "arc195_b", + "ok": false + }, + { + "task_id": "arc195_c", + "ok": false + }, + { + "task_id": "arc195_d", + "ok": false + }, + { + "task_id": "arc195_e", + "ok": false + }, + { + "task_id": "arc196_a", + "ok": false + }, + { + "task_id": "arc196_b", + "ok": false + }, + { + "task_id": "arc196_c", + "ok": false + }, + { + "task_id": "arc196_d", + "ok": false + } + ] + }, + "4": { + "acc": 0.24, + "hard_n": 25, + "hard_acc": 0.36, + "per_item": [ + { + "task_id": "abc382_a", + "ok": true + }, + { + "task_id": "abc382_b", + "ok": true + }, + { + "task_id": "abc382_c", + "ok": false + }, + { + "task_id": "abc382_d", + "ok": false + }, + { + "task_id": "abc382_f", + "ok": false + }, + { + "task_id": "abc382_g", + "ok": false + }, + { + "task_id": "abc383_a", + "ok": true + }, + { + "task_id": "abc383_b", + "ok": true + }, + { + "task_id": "abc383_c", + "ok": false + }, + { + "task_id": "abc383_d", + "ok": false + }, + { + "task_id": "abc383_e", + "ok": false + }, + { + "task_id": "abc384_a", + "ok": true + }, + { + "task_id": "abc384_b", + "ok": true + }, + { + "task_id": "abc384_c", + "ok": true + }, + { + "task_id": "abc384_d", + "ok": false + }, + { + "task_id": "abc384_e", + "ok": false + }, + { + "task_id": "abc384_f", + "ok": false + }, + { + "task_id": "abc384_g", + "ok": false + }, + { + "task_id": "abc385_a", + "ok": false + }, + { + "task_id": "abc385_b", + "ok": true + }, + { + "task_id": "abc385_c", + "ok": false + }, + { + "task_id": "abc385_d", + "ok": false + }, + { + "task_id": "abc385_e", + "ok": false + }, + { + "task_id": "abc385_f", + "ok": false + }, + { + "task_id": "abc386_a", + "ok": false + }, + { + "task_id": "abc386_b", + "ok": false + }, + { + "task_id": "abc386_c", + "ok": false + }, + { + "task_id": "abc386_d", + "ok": false + }, + { + "task_id": "abc386_e", + "ok": false + }, + { + "task_id": "abc386_f", + "ok": false + }, + { + "task_id": "abc387_a", + "ok": true + }, + { + "task_id": "abc387_b", + "ok": true + }, + { + "task_id": "abc387_c", + "ok": false + }, + { + "task_id": "abc387_f", + "ok": false + }, + { + "task_id": "abc388_a", + "ok": true + }, + { + "task_id": "abc388_b", + "ok": true + }, + { + "task_id": "abc388_c", + "ok": false + }, + { + "task_id": "abc388_d", + "ok": false + }, + { + "task_id": "abc388_e", + "ok": false + }, + { + "task_id": "abc388_f", + "ok": false + }, + { + "task_id": "abc388_g", + "ok": false + }, + { + "task_id": "abc389_a", + "ok": true + }, + { + "task_id": "abc389_b", + "ok": true + }, + { + "task_id": "abc389_d", + "ok": false + }, + { + "task_id": "abc389_e", + "ok": false + }, + { + "task_id": "abc389_f", + "ok": false + }, + { + "task_id": "abc389_g", + "ok": false + }, + { + "task_id": "abc390_a", + "ok": true + }, + { + "task_id": "abc390_b", + "ok": false + }, + { + "task_id": "abc390_c", + "ok": true + }, + { + "task_id": "abc390_d", + "ok": false + }, + { + "task_id": "abc390_e", + "ok": false + }, + { + "task_id": "abc390_f", + "ok": false + }, + { + "task_id": "abc390_g", + "ok": false + }, + { + "task_id": "abc391_a", + "ok": true + }, + { + "task_id": "abc391_b", + "ok": true + }, + { + "task_id": "abc391_d", + "ok": false + }, + { + "task_id": "abc391_e", + "ok": false + }, + { + "task_id": "abc391_f", + "ok": false + }, + { + "task_id": "abc391_g", + "ok": false + }, + { + "task_id": "abc392_a", + "ok": true + }, + { + "task_id": "abc392_b", + "ok": true + }, + { + "task_id": "abc392_c", + "ok": false + }, + { + "task_id": "abc392_d", + "ok": false + }, + { + "task_id": "abc392_f", + "ok": false + }, + { + "task_id": "abc392_g", + "ok": false + }, + { + "task_id": "abc393_a", + "ok": true + }, + { + "task_id": "abc393_b", + "ok": true + }, + { + "task_id": "abc393_d", + "ok": false + }, + { + "task_id": "abc393_e", + "ok": false + }, + { + "task_id": "abc393_f", + "ok": false + }, + { + "task_id": "abc394_a", + "ok": true + }, + { + "task_id": "abc394_b", + "ok": true + }, + { + "task_id": "abc394_c", + "ok": true + }, + { + "task_id": "abc394_d", + "ok": false + }, + { + "task_id": "abc394_e", + "ok": false + }, + { + "task_id": "abc394_f", + "ok": false + }, + { + "task_id": "abc394_g", + "ok": false + }, + { + "task_id": "abc395_a", + "ok": true + }, + { + "task_id": "abc395_b", + "ok": true + }, + { + "task_id": "abc395_c", + "ok": true + }, + { + "task_id": "abc395_e", + "ok": false + }, + { + "task_id": "abc395_f", + "ok": false + }, + { + "task_id": "abc396_a", + "ok": true + }, + { + "task_id": "abc396_b", + "ok": true + }, + { + "task_id": "abc396_c", + "ok": false + }, + { + "task_id": "abc396_d", + "ok": false + }, + { + "task_id": "abc396_e", + "ok": false + }, + { + "task_id": "abc396_f", + "ok": false + }, + { + "task_id": "abc396_g", + "ok": false + }, + { + "task_id": "abc397_a", + "ok": true + }, + { + "task_id": "abc397_b", + "ok": false + }, + { + "task_id": "abc397_c", + "ok": false + }, + { + "task_id": "abc397_d", + "ok": false + }, + { + "task_id": "abc397_e", + "ok": false + }, + { + "task_id": "abc397_f", + "ok": false + }, + { + "task_id": "abc397_g", + "ok": false + }, + { + "task_id": "abc398_a", + "ok": false + }, + { + "task_id": "abc398_b", + "ok": true + }, + { + "task_id": "abc398_c", + "ok": true + }, + { + "task_id": "abc398_d", + "ok": false + }, + { + "task_id": "abc398_f", + "ok": false + }, + { + "task_id": "abc398_g", + "ok": false + }, + { + "task_id": "abc399_a", + "ok": true + }, + { + "task_id": "abc399_b", + "ok": true + }, + { + "task_id": "abc399_c", + "ok": false + }, + { + "task_id": "abc399_d", + "ok": false + }, + { + "task_id": "abc399_e", + "ok": false + }, + { + "task_id": "abc399_f", + "ok": false + }, + { + "task_id": "abc400_a", + "ok": true + }, + { + "task_id": "abc400_b", + "ok": false + }, + { + "task_id": "abc400_c", + "ok": false + }, + { + "task_id": "abc400_d", + "ok": false + }, + { + "task_id": "abc400_e", + "ok": false + }, + { + "task_id": "abc400_g", + "ok": false + }, + { + "task_id": "arc188_a", + "ok": false + }, + { + "task_id": "arc188_b", + "ok": false + }, + { + "task_id": "arc188_c", + "ok": false + }, + { + "task_id": "arc188_d", + "ok": false + }, + { + "task_id": "arc189_a", + "ok": false + }, + { + "task_id": "arc189_b", + "ok": false + }, + { + "task_id": "arc189_c", + "ok": false + }, + { + "task_id": "arc189_d", + "ok": false + }, + { + "task_id": "arc190_a", + "ok": false + }, + { + "task_id": "arc190_c", + "ok": false + }, + { + "task_id": "arc190_d", + "ok": false + }, + { + "task_id": "arc191_a", + "ok": false + }, + { + "task_id": "arc191_c", + "ok": false + }, + { + "task_id": "arc191_d", + "ok": false + }, + { + "task_id": "arc192_a", + "ok": false + }, + { + "task_id": "arc192_b", + "ok": false + }, + { + "task_id": "arc192_d", + "ok": false + }, + { + "task_id": "arc192_e", + "ok": false + }, + { + "task_id": "arc193_a", + "ok": false + }, + { + "task_id": "arc193_b", + "ok": false + }, + { + "task_id": "arc193_d", + "ok": false + }, + { + "task_id": "arc194_a", + "ok": false + }, + { + "task_id": "arc194_b", + "ok": false + }, + { + "task_id": "arc194_c", + "ok": false + }, + { + "task_id": "arc194_d", + "ok": false + }, + { + "task_id": "arc194_e", + "ok": false + }, + { + "task_id": "arc195_a", + "ok": false + }, + { + "task_id": "arc195_b", + "ok": false + }, + { + "task_id": "arc195_c", + "ok": false + }, + { + "task_id": "arc195_d", + "ok": false + }, + { + "task_id": "arc195_e", + "ok": false + }, + { + "task_id": "arc196_a", + "ok": false + }, + { + "task_id": "arc196_b", + "ok": false + }, + { + "task_id": "arc196_c", + "ok": false + }, + { + "task_id": "arc196_d", + "ok": false + } + ] + } + } +} \ No newline at end of file diff --git a/results-loop/eval_panel_base_k0.json b/results-loop/eval_panel_base_k0.json new file mode 100644 index 0000000..ea49778 --- /dev/null +++ b/results-loop/eval_panel_base_k0.json @@ -0,0 +1,8 @@ +{ + "tag": "base_k0", + "k": 0, + "arc": 0.36, + "winogrande": 0.55875, + "hellaswag": 0.5225, + "mmlu": 0.30125 +} \ No newline at end of file diff --git a/results-loop/eval_panel_distill_ff.json b/results-loop/eval_panel_distill_ff.json new file mode 100644 index 0000000..6728ff3 --- /dev/null +++ b/results-loop/eval_panel_distill_ff.json @@ -0,0 +1,8 @@ +{ + "tag": "distill_ff", + "k": 1, + "arc": 0.4175, + "winogrande": 0.56625, + "hellaswag": 0.57, + "mmlu": 0.3175 +} \ No newline at end of file diff --git a/results-loop/eval_panel_loop_k2.json b/results-loop/eval_panel_loop_k2.json new file mode 100644 index 0000000..dec0f2e --- /dev/null +++ b/results-loop/eval_panel_loop_k2.json @@ -0,0 +1,8 @@ +{ + "tag": "loop_k2", + "k": 2, + "arc": 0.36125, + "winogrande": 0.5525, + "hellaswag": 0.49625, + "mmlu": 0.3125 +} \ No newline at end of file diff --git a/results-loop/fig_transfer.png b/results-loop/fig_transfer.png index 19e427d..c470b14 100644 Binary files a/results-loop/fig_transfer.png and b/results-loop/fig_transfer.png differ diff --git a/results-loop/lcb_labels.json b/results-loop/lcb_labels.json new file mode 100644 index 0000000..f092297 --- /dev/null +++ b/results-loop/lcb_labels.json @@ -0,0 +1,152 @@ +{ + "abc382_a": true, + "abc382_b": true, + "abc382_c": false, + "abc382_d": true, + "abc382_f": false, + "abc382_g": false, + "abc383_a": true, + "abc383_b": true, + "abc383_c": false, + "abc383_d": true, + "abc383_e": false, + "abc384_a": true, + "abc384_b": true, + "abc384_c": true, + "abc384_d": false, + "abc384_e": false, + "abc384_f": false, + "abc384_g": false, + "abc385_a": true, + "abc385_b": true, + "abc385_c": false, + "abc385_d": false, + "abc385_e": false, + "abc385_f": false, + "abc386_a": false, + "abc386_b": true, + "abc386_c": false, + "abc386_d": false, + "abc386_e": false, + "abc386_f": false, + "abc387_a": true, + "abc387_b": true, + "abc387_c": false, + "abc387_f": false, + "abc388_a": true, + "abc388_b": true, + "abc388_c": false, + "abc388_d": false, + "abc388_e": false, + "abc388_f": false, + "abc388_g": false, + "abc389_a": true, + "abc389_b": true, + "abc389_d": false, + "abc389_e": false, + "abc389_f": false, + "abc389_g": false, + "abc390_a": true, + "abc390_b": true, + "abc390_c": false, + "abc390_d": false, + "abc390_e": false, + "abc390_f": false, + "abc390_g": false, + "abc391_a": true, + "abc391_b": true, + "abc391_d": false, + "abc391_e": false, + "abc391_f": false, + "abc391_g": false, + "abc392_a": true, + "abc392_b": true, + "abc392_c": false, + "abc392_d": false, + "abc392_f": true, + "abc392_g": false, + "abc393_a": true, + "abc393_b": true, + "abc393_d": false, + "abc393_e": true, + "abc393_f": false, + "abc394_a": true, + "abc394_b": true, + "abc394_c": true, + "abc394_d": true, + "abc394_e": false, + "abc394_f": false, + "abc394_g": false, + "abc395_a": true, + "abc395_b": true, + "abc395_c": true, + "abc395_e": true, + "abc395_f": false, + "abc396_a": true, + "abc396_b": true, + "abc396_c": false, + "abc396_d": true, + "abc396_e": false, + "abc396_f": false, + "abc396_g": false, + "abc397_a": true, + "abc397_b": false, + "abc397_c": true, + "abc397_d": false, + "abc397_e": false, + "abc397_f": false, + "abc397_g": false, + "abc398_a": true, + "abc398_b": false, + "abc398_c": true, + "abc398_d": false, + "abc398_f": true, + "abc398_g": false, + "abc399_a": true, + "abc399_b": true, + "abc399_c": true, + "abc399_d": false, + "abc399_e": false, + "abc399_f": false, + "abc400_a": true, + "abc400_b": true, + "abc400_c": false, + "abc400_d": false, + "abc400_e": false, + "abc400_g": false, + "arc188_a": false, + "arc188_b": false, + "arc188_c": false, + "arc188_d": false, + "arc189_a": false, + "arc189_b": false, + "arc189_c": false, + "arc189_d": false, + "arc190_a": false, + "arc190_c": false, + "arc190_d": false, + "arc191_a": false, + "arc191_c": false, + "arc191_d": true, + "arc192_a": false, + "arc192_b": false, + "arc192_d": false, + "arc192_e": false, + "arc193_a": false, + "arc193_b": false, + "arc193_d": false, + "arc194_a": false, + "arc194_b": false, + "arc194_c": false, + "arc194_d": false, + "arc194_e": false, + "arc195_a": false, + "arc195_b": false, + "arc195_c": false, + "arc195_d": false, + "arc195_e": false, + "arc196_a": false, + "arc196_b": false, + "arc196_c": false, + "arc196_d": false +} \ No newline at end of file diff --git a/scripts/figures_final.py b/scripts/figures_final.py index bb48d31..69b4a13 100644 --- a/scripts/figures_final.py +++ b/scripts/figures_final.py @@ -220,8 +220,14 @@ def fig_transfer(): ("base", *ev(OUT / "eval_rust_py_transfer.json", 0)), ("loop k=4", *ev(OUT / "eval_rust_py_transfer.json", 4)), ]), + ("LiveCodeBench (far)\nuntrained wins", [ + ("base", *he_hard("eval_lcb_untrained.json", 0)), + ("untrained k=4", *he_hard("eval_lcb_untrained.json", 4)), + ("loop k=4", *he_hard("eval_lcb_loop.json", 4)), + ("distill", *he_hard("eval_lcb_distill.json", 1)), + ]), ] - fig, axes = plt.subplots(1, 2, figsize=(7.6, 3.8)) + fig, axes = plt.subplots(1, 3, figsize=(10.8, 3.8)) for ax, (title, arms) in zip(axes, groups): style(ax) x = range(len(arms))