diff --git a/PLAN_SELFPACED.md b/PLAN_SELFPACED.md index bc1315c..5c250b9 100644 --- a/PLAN_SELFPACED.md +++ b/PLAN_SELFPACED.md @@ -51,6 +51,56 @@ protection); (c) easy-bucket damage < carry-always's (83→45 was the unprotected number). Hard-bucket *gain* over carry-always is hoped for, not predicted. +### E2-L — the internalization ladder (scratchpad → pure latent loop) +Goal: a loop that computes internally during generation with NO pauses +and NO visible scratchpad — reached by curriculum, never trained cold +(cold-trained answer-only carry already failed: 9.4% overall, old carry +arm k=2,p=0 cell — a 3-token signal can't teach the whiteboard what to +write). Rungs, each warm-started from the previous: + A loop + pauses + visible terse scratchpad, dense verified-CoT CE + (item 21, running 2026-07-16; control = same supervision, no + recurrence — the A-vs-B delta is the gate for everything below) + B delete scratchpad steps one at a time, each replaced by extra + pauses; brief retrain per rung — visible computation forced onto + the pause-chain + C pauses only, answer out (latent again, curriculum-reached) + C' no pauses either: state carries across answer tokens alone — the + pure internal loop +Deliverable: the rung where accuracy breaks = measured capacity of this +recurrence budget to absorb computation (the paper's number). Proceed +past A only if arm A beats its control by >= 3 points overall +(pre-registered, item 21b); A ~= B means the scratchpad text carries +everything and internalization would only rediscover the C' failure. + +### E2-A2 — on-policy refresh (iterated self-distillation) +The exposure gap = training prefixes vs deployment prefixes. Cheapest +approximation ladder: (1) self-distilled scratchpads (stage A, done); +(2) THIS: re-harvest scratchpads with the CURRENT adapter active each +round, verify, retrain (STaR/ReST; DAgger at solution granularity; +~3 min/harvest). Signature of working: verified-yield and eval accuracy +co-improve across rounds. Gated on the A-vs-B verdict. + +### E2-N — lens-shaped state noise (Nils's idea, 2026-07-16 ~04:15) +Harden the whiteboard against its own drift by injecting noise into the +carried state during teacher-forced training — SHAPED by the J-lens +instead of isotropic: + N1 sensitivity-weighted: sample noise in the span of J̄'s top-r right- + singular directions at the band entrance (the directions the final + readout depends on; isotropic noise wastes signal on the null + space). Cheap: jbar.pt exists; --lensnoise rank,scale flag. + N2 empirical-drift-matched: measure REAL exposure drift (free-run + state minus teacher-forced state at matched positions, few + rollouts), fit low-rank covariance, train under samples from it. + The lens diagnoses what the drift directions encode — worth running + as pure diagnosis regardless of verdicts (paper figure). + N3 concept-jitter: lens-read the carried concept (e.g. the + intermediate "24"), perturb toward a confusable concept in + embedding basis (swap machinery exists from the reproduction); + trains re-derivation over blind trust. Most ambitious. +Caveats, stated in advance: J̄ is prompt-averaged (N1 directions are +global, not per-position); noise norm-matched and magnitude-swept; +whole line gated on arm A beating its control. + ### E3 — power knob (only if E1 or E2 shows clean gating) Warm-start rung-2 band-LoRA under the gate; joint fine-tune. Question: do gate-bought iterations do MORE per iteration with a trainable band? diff --git a/results-loop/PROTOCOL_UNIFIED.md b/results-loop/PROTOCOL_UNIFIED.md index d3dc8bc..b186825 100644 --- a/results-loop/PROTOCOL_UNIFIED.md +++ b/results-loop/PROTOCOL_UNIFIED.md @@ -494,3 +494,28 @@ no fixed k solves but some k does. Honest note: arm outputs are VISIBLE tokens (~40) — this is the budget-CoT-with-loop hybrid, a scope change from latent planning, run at Nils's explicit direction ("do gsm8k and such"). + +--- Outcome, item 21 (scored 2026-07-16 ~06:30). Harvest: 427 verified +terse CoTs (292 hard, 135 easy; 73.5% yield, ~3 min). Grid (n=256, +base 10.9/93.1/0.8/0.0 overall/easy/hard/drop): + arm A carry 2:2 53.9 (82.8 / 60.6 / 37.0) + arm A carry 2:6 57.4 (72.4 / 63.8 / 45.0) + control FF 2:2 49.6 (79.3 / 59.8 / 28.0) + control FF 2:6 54.7 (82.8 / 66.9 / 31.0) +(a) CONFIRMED, dramatically: best cell 57.4% vs the previous best GSM +arm's 12.1% — a 5x jump to full-CoT territory (~53%) at ~1/4 the +visible tokens. Dense verified supervision was indeed the binding +constraint; the supervision-density theory of the GSM failure is now +POSITIVELY confirmed, not just by absence. (b) MIXED: A-vs-B delta ++4.3 at p=2 (clears the >=3 bar), +2.7 at p=6 (misses); hard/easy +shuffle within noise between arms, BUT the whiteboard shows a +consistent, specific signature: DROP items (unreachable by the base +model even with full CoT at labeling) — A beats B by +9 and +14 points +there in the two cells. Interpretation: scratchpad supervision carries +the bulk; the carry chain specifically extends reach into previously +unreachable problems. McNemar p-values pending (mcnemar_carrycot.py +ready; classifier-blocked at scoring time). (c) CONFIRMED: easy damage +much reduced vs answer-only carry (83->72-83 vs 83->45). Ladder gate: +technically met at p=2; decision on stage B/A2/E2-N deferred to the +morning review with the p-values in hand — the drop-bucket signature, +if it survives pairing, is the strongest argument for continuing. diff --git a/results-loop/eval_gsm_carrycot_e400.json b/results-loop/eval_gsm_carrycot_e400.json new file mode 100644 index 0000000..2c1b2f2 --- /dev/null +++ b/results-loop/eval_gsm_carrycot_e400.json @@ -0,0 +1,3108 @@ +{ + "tag": "gsm_carrycot_e400", + "grid": { + "0:0": { + "acc": 0.109375, + "by_label": { + "drop": 0.0, + "easy": 0.9310344827586207, + "hard": 0.007874015748031496 + }, + "per_item": [ + { + "idx": 1092, + "ok": false + }, + { + "idx": 696, + "ok": false + }, + { + "idx": 242, + "ok": false + }, + { + "idx": 1085, + "ok": true + }, + { + "idx": 510, + "ok": false + }, + { + "idx": 334, + "ok": true + }, + { + "idx": 139, + "ok": false + }, + { + "idx": 849, + "ok": false + }, + { + "idx": 593, + "ok": false + }, + { + "idx": 579, + "ok": false + }, + { + "idx": 1064, + "ok": false + }, + { + "idx": 275, + "ok": false + }, + { + "idx": 1175, + "ok": false + }, + { + "idx": 1071, + "ok": false + }, + { + "idx": 1284, + "ok": false + }, + { + "idx": 430, + "ok": false + }, + { + "idx": 1088, + "ok": false + }, + { + "idx": 215, + "ok": false + }, + { + "idx": 841, + "ok": false + }, + { + "idx": 1299, + "ok": false + }, + { + "idx": 1113, + "ok": true + }, + { + "idx": 826, + "ok": false + }, + { + "idx": 570, + "ok": false + }, + { + "idx": 598, + "ok": false + }, + { + "idx": 905, + "ok": false + }, + { + "idx": 761, + "ok": false + }, + { + "idx": 1163, + "ok": false + }, + { + "idx": 1287, + "ok": false + }, + { + "idx": 282, + "ok": false + }, + { + "idx": 321, + "ok": false + }, + { + "idx": 252, + "ok": false + }, + { + "idx": 246, + "ok": false + }, + { + "idx": 781, + "ok": false + }, + { + "idx": 821, + "ok": false + }, + { + "idx": 1210, + "ok": true + }, + { + "idx": 958, + "ok": true + }, + { + "idx": 285, + "ok": false + }, + { + "idx": 1146, + "ok": false + }, + { + "idx": 612, + "ok": false + }, + { + "idx": 725, + "ok": false + }, + { + "idx": 1294, + "ok": false + }, + { + "idx": 968, + "ok": false + }, + { + "idx": 850, + "ok": false + }, + { + "idx": 446, + "ok": false + }, + { + "idx": 976, + "ok": false + }, + { + "idx": 1001, + "ok": false + }, + { + "idx": 1027, + "ok": true + }, + { + "idx": 651, + "ok": false + }, + { + "idx": 1009, + "ok": false + }, + { + "idx": 123, + "ok": false + }, + { + "idx": 909, + "ok": false + }, + { + "idx": 614, + "ok": false + }, + { + "idx": 292, + "ok": false + }, + { + "idx": 1014, + "ok": false + }, + { + "idx": 107, + "ok": false + }, + { + "idx": 1272, + "ok": false + }, + { + "idx": 441, + "ok": false + }, + { + "idx": 52, + "ok": false + }, + { + "idx": 727, + "ok": false + }, + { + "idx": 965, + "ok": false + }, + { + "idx": 800, + "ok": false + }, + { + "idx": 20, + "ok": false + }, + { + "idx": 1078, + "ok": false + }, + { + "idx": 136, + "ok": false + }, + { + "idx": 166, + "ok": true + }, + { + "idx": 809, + "ok": true + }, + { + "idx": 13, + "ok": false + }, + { + "idx": 739, + "ok": false + }, + { + "idx": 84, + "ok": true + }, + { + "idx": 237, + "ok": true + }, + { + "idx": 1271, + "ok": false + }, + { + "idx": 7, + "ok": false + }, + { + "idx": 553, + "ok": false + }, + { + "idx": 1311, + "ok": false + }, + { + "idx": 599, + "ok": false + }, + { + "idx": 464, + "ok": false + }, + { + "idx": 288, + "ok": false + }, + { + "idx": 1173, + "ok": false + }, + { + "idx": 589, + "ok": false + }, + { + "idx": 391, + "ok": false + }, + { + "idx": 888, + "ok": true + }, + { + "idx": 943, + "ok": false + }, + { + "idx": 676, + "ok": false + }, + { + "idx": 786, + "ok": false + }, + { + "idx": 344, + "ok": true + }, + { + "idx": 677, + "ok": false + }, + { + "idx": 863, + "ok": false + }, + { + "idx": 891, + "ok": false + }, + { + "idx": 303, + "ok": false + }, + { + "idx": 916, + "ok": false + }, + { + "idx": 302, + "ok": false + }, + { + "idx": 1073, + "ok": false + }, + { + "idx": 647, + "ok": false + }, + { + "idx": 264, + "ok": false + }, + { + "idx": 427, + "ok": false + }, + { + "idx": 382, + "ok": false + }, + { + "idx": 715, + "ok": false + }, + { + "idx": 796, + "ok": false + }, + { + "idx": 875, + "ok": false + }, + { + "idx": 1007, + "ok": false + }, + { + "idx": 797, + "ok": true + }, + { + "idx": 450, + "ok": false + }, + { + "idx": 401, + "ok": false + }, + { + "idx": 899, + "ok": false + }, + { + "idx": 418, + "ok": false + }, + { + "idx": 1201, + "ok": false + }, + { + "idx": 101, + "ok": false + }, + { + "idx": 795, + "ok": false + }, + { + "idx": 68, + "ok": false + }, + { + "idx": 479, + "ok": false + }, + { + "idx": 1296, + "ok": false + }, + { + "idx": 173, + "ok": false + }, + { + "idx": 744, + "ok": false + }, + { + "idx": 116, + "ok": true + }, + { + "idx": 1306, + "ok": false + }, + { + "idx": 355, + "ok": false + }, + { + "idx": 477, + "ok": false + }, + { + "idx": 1250, + "ok": false + }, + { + "idx": 609, + "ok": false + }, + { + "idx": 1254, + "ok": true + }, + { + "idx": 177, + "ok": false + }, + { + "idx": 1048, + "ok": false + }, + { + "idx": 582, + "ok": false + }, + { + "idx": 723, + "ok": false + }, + { + "idx": 842, + "ok": false + }, + { + "idx": 938, + "ok": false + }, + { + "idx": 110, + "ok": false + }, + { + "idx": 1057, + "ok": false + }, + { + "idx": 1120, + "ok": false + }, + { + "idx": 881, + "ok": false + }, + { + "idx": 1189, + "ok": false + }, + { + "idx": 931, + "ok": false + }, + { + "idx": 1003, + "ok": false + }, + { + "idx": 521, + "ok": false + }, + { + "idx": 974, + "ok": false + }, + { + "idx": 690, + "ok": false + }, + { + "idx": 544, + "ok": false + }, + { + "idx": 86, + "ok": false + }, + { + "idx": 89, + "ok": false + }, + { + "idx": 333, + "ok": false + }, + { + "idx": 716, + "ok": false + }, + { + "idx": 14, + "ok": false + }, + { + "idx": 287, + "ok": false + }, + { + "idx": 130, + "ok": false + }, + { + "idx": 876, + "ok": false + }, + { + "idx": 454, + "ok": false + }, + { + "idx": 1246, + "ok": true + }, + { + "idx": 812, + "ok": false + }, + { + "idx": 1142, + "ok": false + }, + { + "idx": 452, + "ok": false + }, + { + "idx": 929, + "ok": false + }, + { + "idx": 394, + "ok": false + }, + { + "idx": 695, + "ok": false + }, + { + "idx": 1245, + "ok": false + }, + { + "idx": 209, + "ok": false + }, + { + "idx": 1241, + "ok": false + }, + { + "idx": 175, + "ok": false + }, + { + "idx": 653, + "ok": false + }, + { + "idx": 661, + "ok": false + }, + { + "idx": 1097, + "ok": false + }, + { + "idx": 933, + "ok": false + }, + { + "idx": 665, + "ok": false + }, + { + "idx": 523, + "ok": false + }, + { + "idx": 59, + "ok": false + }, + { + "idx": 1069, + "ok": false + }, + { + "idx": 90, + "ok": false + }, + { + "idx": 389, + "ok": false + }, + { + "idx": 755, + "ok": false + }, + { + "idx": 164, + "ok": false + }, + { + "idx": 429, + "ok": false + }, + { + "idx": 1074, + "ok": false + }, + { + "idx": 708, + "ok": true + }, + { + "idx": 384, + "ok": true + }, + { + "idx": 412, + "ok": false + }, + { + "idx": 514, + "ok": false + }, + { + "idx": 617, + "ok": false + }, + { + "idx": 639, + "ok": false + }, + { + "idx": 1058, + "ok": false + }, + { + "idx": 787, + "ok": false + }, + { + "idx": 987, + "ok": false + }, + { + "idx": 704, + "ok": false + }, + { + "idx": 491, + "ok": false + }, + { + "idx": 91, + "ok": false + }, + { + "idx": 626, + "ok": false + }, + { + "idx": 1130, + "ok": false + }, + { + "idx": 148, + "ok": false + }, + { + "idx": 18, + "ok": false + }, + { + "idx": 897, + "ok": true + }, + { + "idx": 97, + "ok": false + }, + { + "idx": 844, + "ok": false + }, + { + "idx": 1010, + "ok": false + }, + { + "idx": 901, + "ok": false + }, + { + "idx": 241, + "ok": false + }, + { + "idx": 493, + "ok": false + }, + { + "idx": 202, + "ok": true + }, + { + "idx": 314, + "ok": false + }, + { + "idx": 847, + "ok": false + }, + { + "idx": 437, + "ok": false + }, + { + "idx": 902, + "ok": true + }, + { + "idx": 157, + "ok": false + }, + { + "idx": 874, + "ok": true + }, + { + "idx": 1144, + "ok": false + }, + { + "idx": 807, + "ok": false + }, + { + "idx": 80, + "ok": true + }, + { + "idx": 369, + "ok": false + }, + { + "idx": 511, + "ok": false + }, + { + "idx": 262, + "ok": false + }, + { + "idx": 571, + "ok": false + }, + { + "idx": 721, + "ok": false + }, + { + "idx": 654, + "ok": false + }, + { + "idx": 890, + "ok": true + }, + { + "idx": 219, + "ok": false + }, + { + "idx": 1140, + "ok": false + }, + { + "idx": 584, + "ok": false + }, + { + "idx": 1249, + "ok": false + }, + { + "idx": 1108, + "ok": true + }, + { + "idx": 607, + "ok": false + }, + { + "idx": 904, + "ok": false + }, + { + "idx": 1052, + "ok": false + }, + { + "idx": 1238, + "ok": false + }, + { + "idx": 946, + "ok": false + }, + { + "idx": 1098, + "ok": false + }, + { + "idx": 534, + "ok": false + }, + { + "idx": 558, + "ok": false + }, + { + "idx": 474, + "ok": true + }, + { + "idx": 33, + "ok": false + }, + { + "idx": 243, + "ok": false + }, + { + "idx": 1259, + "ok": false + }, + { + "idx": 353, + "ok": false + }, + { + "idx": 509, + "ok": false + }, + { + "idx": 1096, + "ok": false + }, + { + "idx": 1055, + "ok": false + }, + { + "idx": 877, + "ok": false + }, + { + "idx": 249, + "ok": false + }, + { + "idx": 1156, + "ok": false + }, + { + "idx": 735, + "ok": false + }, + { + "idx": 470, + "ok": false + }, + { + "idx": 1119, + "ok": false + }, + { + "idx": 577, + "ok": true + }, + { + "idx": 453, + "ok": false + }, + { + "idx": 132, + "ok": false + }, + { + "idx": 1061, + "ok": false + }, + { + "idx": 629, + "ok": false + }, + { + "idx": 670, + "ok": true + }, + { + "idx": 478, + "ok": false + }, + { + "idx": 764, + "ok": false + }, + { + "idx": 983, + "ok": false + }, + { + "idx": 587, + "ok": false + }, + { + "idx": 1192, + "ok": false + }, + { + "idx": 350, + "ok": false + }, + { + "idx": 281, + "ok": false + }, + { + "idx": 31, + "ok": false + }, + { + "idx": 1132, + "ok": false + }, + { + "idx": 1033, + "ok": false + }, + { + "idx": 671, + "ok": false + }, + { + "idx": 751, + "ok": false + } + ] + }, + "2:2": { + "acc": 0.5390625, + "by_label": { + "drop": 0.37, + "easy": 0.8275862068965517, + "hard": 0.6062992125984252 + }, + "per_item": [ + { + "idx": 1092, + "ok": false + }, + { + "idx": 696, + "ok": false + }, + { + "idx": 242, + "ok": false + }, + { + "idx": 1085, + "ok": true + }, + { + "idx": 510, + "ok": false + }, + { + "idx": 334, + "ok": true + }, + { + "idx": 139, + "ok": true + }, + { + "idx": 849, + "ok": false + }, + { + "idx": 593, + "ok": true + }, + { + "idx": 579, + "ok": true + }, + { + "idx": 1064, + "ok": false + }, + { + "idx": 275, + "ok": false + }, + { + "idx": 1175, + "ok": false + }, + { + "idx": 1071, + "ok": false + }, + { + "idx": 1284, + "ok": false + }, + { + "idx": 430, + "ok": true + }, + { + "idx": 1088, + "ok": false + }, + { + "idx": 215, + "ok": true + }, + { + "idx": 841, + "ok": true + }, + { + "idx": 1299, + "ok": false + }, + { + "idx": 1113, + "ok": true + }, + { + "idx": 826, + "ok": false + }, + { + "idx": 570, + "ok": false + }, + { + "idx": 598, + "ok": false + }, + { + "idx": 905, + "ok": false + }, + { + "idx": 761, + "ok": true + }, + { + "idx": 1163, + "ok": true + }, + { + "idx": 1287, + "ok": false + }, + { + "idx": 282, + "ok": false + }, + { + "idx": 321, + "ok": false + }, + { + "idx": 252, + "ok": false + }, + { + "idx": 246, + "ok": false + }, + { + "idx": 781, + "ok": false + }, + { + "idx": 821, + "ok": true + }, + { + "idx": 1210, + "ok": true + }, + { + "idx": 958, + "ok": false + }, + { + "idx": 285, + "ok": true + }, + { + "idx": 1146, + "ok": true + }, + { + "idx": 612, + "ok": true + }, + { + "idx": 725, + "ok": true + }, + { + "idx": 1294, + "ok": false + }, + { + "idx": 968, + "ok": false + }, + { + "idx": 850, + "ok": false + }, + { + "idx": 446, + "ok": false + }, + { + "idx": 976, + "ok": false + }, + { + "idx": 1001, + "ok": false + }, + { + "idx": 1027, + "ok": true + }, + { + "idx": 651, + "ok": true + }, + { + "idx": 1009, + "ok": false + }, + { + "idx": 123, + "ok": true + }, + { + "idx": 909, + "ok": true + }, + { + "idx": 614, + "ok": false + }, + { + "idx": 292, + "ok": true + }, + { + "idx": 1014, + "ok": false + }, + { + "idx": 107, + "ok": true + }, + { + "idx": 1272, + "ok": false + }, + { + "idx": 441, + "ok": true + }, + { + "idx": 52, + "ok": true + }, + { + "idx": 727, + "ok": false + }, + { + "idx": 965, + "ok": true + }, + { + "idx": 800, + "ok": true + }, + { + "idx": 20, + "ok": false + }, + { + "idx": 1078, + "ok": true + }, + { + "idx": 136, + "ok": true + }, + { + "idx": 166, + "ok": true + }, + { + "idx": 809, + "ok": true + }, + { + "idx": 13, + "ok": false + }, + { + "idx": 739, + "ok": true + }, + { + "idx": 84, + "ok": true + }, + { + "idx": 237, + "ok": true + }, + { + "idx": 1271, + "ok": false + }, + { + "idx": 7, + "ok": true + }, + { + "idx": 553, + "ok": true + }, + { + "idx": 1311, + "ok": false + }, + { + "idx": 599, + "ok": true + }, + { + "idx": 464, + "ok": false + }, + { + "idx": 288, + "ok": false + }, + { + "idx": 1173, + "ok": false + }, + { + "idx": 589, + "ok": false + }, + { + "idx": 391, + "ok": true + }, + { + "idx": 888, + "ok": true + }, + { + "idx": 943, + "ok": false + }, + { + "idx": 676, + "ok": true + }, + { + "idx": 786, + "ok": true + }, + { + "idx": 344, + "ok": true + }, + { + "idx": 677, + "ok": true + }, + { + "idx": 863, + "ok": true + }, + { + "idx": 891, + "ok": false + }, + { + "idx": 303, + "ok": false + }, + { + "idx": 916, + "ok": false + }, + { + "idx": 302, + "ok": true + }, + { + "idx": 1073, + "ok": false + }, + { + "idx": 647, + "ok": false + }, + { + "idx": 264, + "ok": true + }, + { + "idx": 427, + "ok": false + }, + { + "idx": 382, + "ok": false + }, + { + "idx": 715, + "ok": true + }, + { + "idx": 796, + "ok": false + }, + { + "idx": 875, + "ok": true + }, + { + "idx": 1007, + "ok": false + }, + { + "idx": 797, + "ok": true + }, + { + "idx": 450, + "ok": true + }, + { + "idx": 401, + "ok": false + }, + { + "idx": 899, + "ok": true + }, + { + "idx": 418, + "ok": false + }, + { + "idx": 1201, + "ok": true + }, + { + "idx": 101, + "ok": false + }, + { + "idx": 795, + "ok": true + }, + { + "idx": 68, + "ok": true + }, + { + "idx": 479, + "ok": false + }, + { + "idx": 1296, + "ok": true + }, + { + "idx": 173, + "ok": false + }, + { + "idx": 744, + "ok": true + }, + { + "idx": 116, + "ok": true + }, + { + "idx": 1306, + "ok": false + }, + { + "idx": 355, + "ok": true + }, + { + "idx": 477, + "ok": true + }, + { + "idx": 1250, + "ok": true + }, + { + "idx": 609, + "ok": true + }, + { + "idx": 1254, + "ok": true + }, + { + "idx": 177, + "ok": false + }, + { + "idx": 1048, + "ok": false + }, + { + "idx": 582, + "ok": false + }, + { + "idx": 723, + "ok": true + }, + { + "idx": 842, + "ok": true + }, + { + "idx": 938, + "ok": true + }, + { + "idx": 110, + "ok": true + }, + { + "idx": 1057, + "ok": true + }, + { + "idx": 1120, + "ok": true + }, + { + "idx": 881, + "ok": true + }, + { + "idx": 1189, + "ok": false + }, + { + "idx": 931, + "ok": true + }, + { + "idx": 1003, + "ok": true + }, + { + "idx": 521, + "ok": false + }, + { + "idx": 974, + "ok": true + }, + { + "idx": 690, + "ok": false + }, + { + "idx": 544, + "ok": true + }, + { + "idx": 86, + "ok": true + }, + { + "idx": 89, + "ok": true + }, + { + "idx": 333, + "ok": true + }, + { + "idx": 716, + "ok": true + }, + { + "idx": 14, + "ok": true + }, + { + "idx": 287, + "ok": false + }, + { + "idx": 130, + "ok": false + }, + { + "idx": 876, + "ok": true + }, + { + "idx": 454, + "ok": false + }, + { + "idx": 1246, + "ok": true + }, + { + "idx": 812, + "ok": false + }, + { + "idx": 1142, + "ok": true + }, + { + "idx": 452, + "ok": true + }, + { + "idx": 929, + "ok": true + }, + { + "idx": 394, + "ok": true + }, + { + "idx": 695, + "ok": true + }, + { + "idx": 1245, + "ok": true + }, + { + "idx": 209, + "ok": false + }, + { + "idx": 1241, + "ok": false + }, + { + "idx": 175, + "ok": true + }, + { + "idx": 653, + "ok": false + }, + { + "idx": 661, + "ok": true + }, + { + "idx": 1097, + "ok": true + }, + { + "idx": 933, + "ok": true + }, + { + "idx": 665, + "ok": true + }, + { + "idx": 523, + "ok": true + }, + { + "idx": 59, + "ok": true + }, + { + "idx": 1069, + "ok": true + }, + { + "idx": 90, + "ok": false + }, + { + "idx": 389, + "ok": false + }, + { + "idx": 755, + "ok": false + }, + { + "idx": 164, + "ok": true + }, + { + "idx": 429, + "ok": true + }, + { + "idx": 1074, + "ok": false + }, + { + "idx": 708, + "ok": true + }, + { + "idx": 384, + "ok": false + }, + { + "idx": 412, + "ok": true + }, + { + "idx": 514, + "ok": true + }, + { + "idx": 617, + "ok": true + }, + { + "idx": 639, + "ok": true + }, + { + "idx": 1058, + "ok": false + }, + { + "idx": 787, + "ok": true + }, + { + "idx": 987, + "ok": true + }, + { + "idx": 704, + "ok": true + }, + { + "idx": 491, + "ok": false + }, + { + "idx": 91, + "ok": true + }, + { + "idx": 626, + "ok": true + }, + { + "idx": 1130, + "ok": false + }, + { + "idx": 148, + "ok": false + }, + { + "idx": 18, + "ok": false + }, + { + "idx": 897, + "ok": false + }, + { + "idx": 97, + "ok": false + }, + { + "idx": 844, + "ok": true + }, + { + "idx": 1010, + "ok": true + }, + { + "idx": 901, + "ok": false + }, + { + "idx": 241, + "ok": false + }, + { + "idx": 493, + "ok": false + }, + { + "idx": 202, + "ok": true + }, + { + "idx": 314, + "ok": true + }, + { + "idx": 847, + "ok": false + }, + { + "idx": 437, + "ok": true + }, + { + "idx": 902, + "ok": true + }, + { + "idx": 157, + "ok": false + }, + { + "idx": 874, + "ok": true + }, + { + "idx": 1144, + "ok": true + }, + { + "idx": 807, + "ok": false + }, + { + "idx": 80, + "ok": false + }, + { + "idx": 369, + "ok": false + }, + { + "idx": 511, + "ok": true + }, + { + "idx": 262, + "ok": true + }, + { + "idx": 571, + "ok": false + }, + { + "idx": 721, + "ok": false + }, + { + "idx": 654, + "ok": false + }, + { + "idx": 890, + "ok": false + }, + { + "idx": 219, + "ok": false + }, + { + "idx": 1140, + "ok": false + }, + { + "idx": 584, + "ok": false + }, + { + "idx": 1249, + "ok": true + }, + { + "idx": 1108, + "ok": true + }, + { + "idx": 607, + "ok": false + }, + { + "idx": 904, + "ok": true + }, + { + "idx": 1052, + "ok": false + }, + { + "idx": 1238, + "ok": true + }, + { + "idx": 946, + "ok": false + }, + { + "idx": 1098, + "ok": true + }, + { + "idx": 534, + "ok": false + }, + { + "idx": 558, + "ok": true + }, + { + "idx": 474, + "ok": true + }, + { + "idx": 33, + "ok": false + }, + { + "idx": 243, + "ok": false + }, + { + "idx": 1259, + "ok": false + }, + { + "idx": 353, + "ok": false + }, + { + "idx": 509, + "ok": false + }, + { + "idx": 1096, + "ok": false + }, + { + "idx": 1055, + "ok": true + }, + { + "idx": 877, + "ok": true + }, + { + "idx": 249, + "ok": true + }, + { + "idx": 1156, + "ok": false + }, + { + "idx": 735, + "ok": true + }, + { + "idx": 470, + "ok": false + }, + { + "idx": 1119, + "ok": false + }, + { + "idx": 577, + "ok": false + }, + { + "idx": 453, + "ok": false + }, + { + "idx": 132, + "ok": false + }, + { + "idx": 1061, + "ok": true + }, + { + "idx": 629, + "ok": true + }, + { + "idx": 670, + "ok": true + }, + { + "idx": 478, + "ok": false + }, + { + "idx": 764, + "ok": true + }, + { + "idx": 983, + "ok": true + }, + { + "idx": 587, + "ok": true + }, + { + "idx": 1192, + "ok": true + }, + { + "idx": 350, + "ok": true + }, + { + "idx": 281, + "ok": false + }, + { + "idx": 31, + "ok": false + }, + { + "idx": 1132, + "ok": false + }, + { + "idx": 1033, + "ok": true + }, + { + "idx": 671, + "ok": true + }, + { + "idx": 751, + "ok": true + } + ] + }, + "2:6": { + "acc": 0.57421875, + "by_label": { + "drop": 0.45, + "easy": 0.7241379310344828, + "hard": 0.6377952755905512 + }, + "per_item": [ + { + "idx": 1092, + "ok": false + }, + { + "idx": 696, + "ok": false + }, + { + "idx": 242, + "ok": false + }, + { + "idx": 1085, + "ok": true + }, + { + "idx": 510, + "ok": false + }, + { + "idx": 334, + "ok": true + }, + { + "idx": 139, + "ok": true + }, + { + "idx": 849, + "ok": true + }, + { + "idx": 593, + "ok": true + }, + { + "idx": 579, + "ok": true + }, + { + "idx": 1064, + "ok": false + }, + { + "idx": 275, + "ok": false + }, + { + "idx": 1175, + "ok": false + }, + { + "idx": 1071, + "ok": false + }, + { + "idx": 1284, + "ok": true + }, + { + "idx": 430, + "ok": true + }, + { + "idx": 1088, + "ok": false + }, + { + "idx": 215, + "ok": true + }, + { + "idx": 841, + "ok": false + }, + { + "idx": 1299, + "ok": false + }, + { + "idx": 1113, + "ok": true + }, + { + "idx": 826, + "ok": false + }, + { + "idx": 570, + "ok": false + }, + { + "idx": 598, + "ok": false + }, + { + "idx": 905, + "ok": false + }, + { + "idx": 761, + "ok": false + }, + { + "idx": 1163, + "ok": true + }, + { + "idx": 1287, + "ok": true + }, + { + "idx": 282, + "ok": false + }, + { + "idx": 321, + "ok": true + }, + { + "idx": 252, + "ok": false + }, + { + "idx": 246, + "ok": true + }, + { + "idx": 781, + "ok": false + }, + { + "idx": 821, + "ok": true + }, + { + "idx": 1210, + "ok": false + }, + { + "idx": 958, + "ok": false + }, + { + "idx": 285, + "ok": true + }, + { + "idx": 1146, + "ok": true + }, + { + "idx": 612, + "ok": true + }, + { + "idx": 725, + "ok": true + }, + { + "idx": 1294, + "ok": false + }, + { + "idx": 968, + "ok": true + }, + { + "idx": 850, + "ok": true + }, + { + "idx": 446, + "ok": false + }, + { + "idx": 976, + "ok": false + }, + { + "idx": 1001, + "ok": false + }, + { + "idx": 1027, + "ok": true + }, + { + "idx": 651, + "ok": true + }, + { + "idx": 1009, + "ok": false + }, + { + "idx": 123, + "ok": true + }, + { + "idx": 909, + "ok": true + }, + { + "idx": 614, + "ok": false + }, + { + "idx": 292, + "ok": true + }, + { + "idx": 1014, + "ok": false + }, + { + "idx": 107, + "ok": true + }, + { + "idx": 1272, + "ok": true + }, + { + "idx": 441, + "ok": true + }, + { + "idx": 52, + "ok": true + }, + { + "idx": 727, + "ok": false + }, + { + "idx": 965, + "ok": true + }, + { + "idx": 800, + "ok": true + }, + { + "idx": 20, + "ok": false + }, + { + "idx": 1078, + "ok": true + }, + { + "idx": 136, + "ok": true + }, + { + "idx": 166, + "ok": true + }, + { + "idx": 809, + "ok": false + }, + { + "idx": 13, + "ok": false + }, + { + "idx": 739, + "ok": true + }, + { + "idx": 84, + "ok": true + }, + { + "idx": 237, + "ok": true + }, + { + "idx": 1271, + "ok": true + }, + { + "idx": 7, + "ok": true + }, + { + "idx": 553, + "ok": true + }, + { + "idx": 1311, + "ok": false + }, + { + "idx": 599, + "ok": true + }, + { + "idx": 464, + "ok": false + }, + { + "idx": 288, + "ok": true + }, + { + "idx": 1173, + "ok": true + }, + { + "idx": 589, + "ok": false + }, + { + "idx": 391, + "ok": false + }, + { + "idx": 888, + "ok": true + }, + { + "idx": 943, + "ok": false + }, + { + "idx": 676, + "ok": true + }, + { + "idx": 786, + "ok": true + }, + { + "idx": 344, + "ok": true + }, + { + "idx": 677, + "ok": true + }, + { + "idx": 863, + "ok": false + }, + { + "idx": 891, + "ok": true + }, + { + "idx": 303, + "ok": false + }, + { + "idx": 916, + "ok": true + }, + { + "idx": 302, + "ok": true + }, + { + "idx": 1073, + "ok": false + }, + { + "idx": 647, + "ok": false + }, + { + "idx": 264, + "ok": true + }, + { + "idx": 427, + "ok": false + }, + { + "idx": 382, + "ok": false + }, + { + "idx": 715, + "ok": true + }, + { + "idx": 796, + "ok": false + }, + { + "idx": 875, + "ok": true + }, + { + "idx": 1007, + "ok": false + }, + { + "idx": 797, + "ok": true + }, + { + "idx": 450, + "ok": true + }, + { + "idx": 401, + "ok": false + }, + { + "idx": 899, + "ok": true + }, + { + "idx": 418, + "ok": false + }, + { + "idx": 1201, + "ok": true + }, + { + "idx": 101, + "ok": false + }, + { + "idx": 795, + "ok": true + }, + { + "idx": 68, + "ok": false + }, + { + "idx": 479, + "ok": false + }, + { + "idx": 1296, + "ok": true + }, + { + "idx": 173, + "ok": true + }, + { + "idx": 744, + "ok": true + }, + { + "idx": 116, + "ok": true + }, + { + "idx": 1306, + "ok": false + }, + { + "idx": 355, + "ok": true + }, + { + "idx": 477, + "ok": false + }, + { + "idx": 1250, + "ok": true + }, + { + "idx": 609, + "ok": true + }, + { + "idx": 1254, + "ok": true + }, + { + "idx": 177, + "ok": false + }, + { + "idx": 1048, + "ok": false + }, + { + "idx": 582, + "ok": true + }, + { + "idx": 723, + "ok": true + }, + { + "idx": 842, + "ok": true + }, + { + "idx": 938, + "ok": true + }, + { + "idx": 110, + "ok": true + }, + { + "idx": 1057, + "ok": false + }, + { + "idx": 1120, + "ok": true + }, + { + "idx": 881, + "ok": false + }, + { + "idx": 1189, + "ok": false + }, + { + "idx": 931, + "ok": false + }, + { + "idx": 1003, + "ok": false + }, + { + "idx": 521, + "ok": false + }, + { + "idx": 974, + "ok": true + }, + { + "idx": 690, + "ok": false + }, + { + "idx": 544, + "ok": true + }, + { + "idx": 86, + "ok": true + }, + { + "idx": 89, + "ok": true + }, + { + "idx": 333, + "ok": true + }, + { + "idx": 716, + "ok": true + }, + { + "idx": 14, + "ok": true + }, + { + "idx": 287, + "ok": true + }, + { + "idx": 130, + "ok": true + }, + { + "idx": 876, + "ok": true + }, + { + "idx": 454, + "ok": false + }, + { + "idx": 1246, + "ok": true + }, + { + "idx": 812, + "ok": false + }, + { + "idx": 1142, + "ok": true + }, + { + "idx": 452, + "ok": true + }, + { + "idx": 929, + "ok": true + }, + { + "idx": 394, + "ok": true + }, + { + "idx": 695, + "ok": true + }, + { + "idx": 1245, + "ok": true + }, + { + "idx": 209, + "ok": false + }, + { + "idx": 1241, + "ok": false + }, + { + "idx": 175, + "ok": true + }, + { + "idx": 653, + "ok": false + }, + { + "idx": 661, + "ok": true + }, + { + "idx": 1097, + "ok": true + }, + { + "idx": 933, + "ok": true + }, + { + "idx": 665, + "ok": true + }, + { + "idx": 523, + "ok": false + }, + { + "idx": 59, + "ok": true + }, + { + "idx": 1069, + "ok": true + }, + { + "idx": 90, + "ok": true + }, + { + "idx": 389, + "ok": false + }, + { + "idx": 755, + "ok": false + }, + { + "idx": 164, + "ok": true + }, + { + "idx": 429, + "ok": true + }, + { + "idx": 1074, + "ok": true + }, + { + "idx": 708, + "ok": true + }, + { + "idx": 384, + "ok": true + }, + { + "idx": 412, + "ok": true + }, + { + "idx": 514, + "ok": true + }, + { + "idx": 617, + "ok": true + }, + { + "idx": 639, + "ok": true + }, + { + "idx": 1058, + "ok": false + }, + { + "idx": 787, + "ok": true + }, + { + "idx": 987, + "ok": true + }, + { + "idx": 704, + "ok": true + }, + { + "idx": 491, + "ok": true + }, + { + "idx": 91, + "ok": false + }, + { + "idx": 626, + "ok": true + }, + { + "idx": 1130, + "ok": false + }, + { + "idx": 148, + "ok": false + }, + { + "idx": 18, + "ok": false + }, + { + "idx": 897, + "ok": false + }, + { + "idx": 97, + "ok": true + }, + { + "idx": 844, + "ok": false + }, + { + "idx": 1010, + "ok": true + }, + { + "idx": 901, + "ok": false + }, + { + "idx": 241, + "ok": false + }, + { + "idx": 493, + "ok": false + }, + { + "idx": 202, + "ok": true + }, + { + "idx": 314, + "ok": true + }, + { + "idx": 847, + "ok": true + }, + { + "idx": 437, + "ok": true + }, + { + "idx": 902, + "ok": true + }, + { + "idx": 157, + "ok": false + }, + { + "idx": 874, + "ok": true + }, + { + "idx": 1144, + "ok": true + }, + { + "idx": 807, + "ok": false + }, + { + "idx": 80, + "ok": false + }, + { + "idx": 369, + "ok": false + }, + { + "idx": 511, + "ok": true + }, + { + "idx": 262, + "ok": true + }, + { + "idx": 571, + "ok": true + }, + { + "idx": 721, + "ok": true + }, + { + "idx": 654, + "ok": false + }, + { + "idx": 890, + "ok": false + }, + { + "idx": 219, + "ok": false + }, + { + "idx": 1140, + "ok": false + }, + { + "idx": 584, + "ok": false + }, + { + "idx": 1249, + "ok": true + }, + { + "idx": 1108, + "ok": true + }, + { + "idx": 607, + "ok": false + }, + { + "idx": 904, + "ok": true + }, + { + "idx": 1052, + "ok": false + }, + { + "idx": 1238, + "ok": true + }, + { + "idx": 946, + "ok": false + }, + { + "idx": 1098, + "ok": true + }, + { + "idx": 534, + "ok": false + }, + { + "idx": 558, + "ok": true + }, + { + "idx": 474, + "ok": false + }, + { + "idx": 33, + "ok": true + }, + { + "idx": 243, + "ok": true + }, + { + "idx": 1259, + "ok": false + }, + { + "idx": 353, + "ok": false + }, + { + "idx": 509, + "ok": false + }, + { + "idx": 1096, + "ok": true + }, + { + "idx": 1055, + "ok": true + }, + { + "idx": 877, + "ok": true + }, + { + "idx": 249, + "ok": false + }, + { + "idx": 1156, + "ok": false + }, + { + "idx": 735, + "ok": true + }, + { + "idx": 470, + "ok": false + }, + { + "idx": 1119, + "ok": false + }, + { + "idx": 577, + "ok": false + }, + { + "idx": 453, + "ok": false + }, + { + "idx": 132, + "ok": false + }, + { + "idx": 1061, + "ok": true + }, + { + "idx": 629, + "ok": true + }, + { + "idx": 670, + "ok": true + }, + { + "idx": 478, + "ok": false + }, + { + "idx": 764, + "ok": true + }, + { + "idx": 983, + "ok": true + }, + { + "idx": 587, + "ok": true + }, + { + "idx": 1192, + "ok": false + }, + { + "idx": 350, + "ok": false + }, + { + "idx": 281, + "ok": false + }, + { + "idx": 31, + "ok": false + }, + { + "idx": 1132, + "ok": false + }, + { + "idx": 1033, + "ok": true + }, + { + "idx": 671, + "ok": true + }, + { + "idx": 751, + "ok": true + } + ] + } + }, + "n": 256 +} \ No newline at end of file diff --git a/results-loop/eval_gsm_carrycot_ff_e400.json b/results-loop/eval_gsm_carrycot_ff_e400.json new file mode 100644 index 0000000..66b36d8 --- /dev/null +++ b/results-loop/eval_gsm_carrycot_ff_e400.json @@ -0,0 +1,2074 @@ +{ + "tag": "gsm_carrycot_ff_e400", + "grid": { + "2:2": { + "acc": 0.49609375, + "by_label": { + "drop": 0.28, + "easy": 0.7931034482758621, + "hard": 0.5984251968503937 + }, + "per_item": [ + { + "idx": 1092, + "ok": false + }, + { + "idx": 696, + "ok": true + }, + { + "idx": 242, + "ok": false + }, + { + "idx": 1085, + "ok": true + }, + { + "idx": 510, + "ok": false + }, + { + "idx": 334, + "ok": true + }, + { + "idx": 139, + "ok": true + }, + { + "idx": 849, + "ok": true + }, + { + "idx": 593, + "ok": true + }, + { + "idx": 579, + "ok": true + }, + { + "idx": 1064, + "ok": false + }, + { + "idx": 275, + "ok": false + }, + { + "idx": 1175, + "ok": false + }, + { + "idx": 1071, + "ok": false + }, + { + "idx": 1284, + "ok": false + }, + { + "idx": 430, + "ok": false + }, + { + "idx": 1088, + "ok": false + }, + { + "idx": 215, + "ok": true + }, + { + "idx": 841, + "ok": false + }, + { + "idx": 1299, + "ok": true + }, + { + "idx": 1113, + "ok": true + }, + { + "idx": 826, + "ok": true + }, + { + "idx": 570, + "ok": false + }, + { + "idx": 598, + "ok": true + }, + { + "idx": 905, + "ok": false + }, + { + "idx": 761, + "ok": true + }, + { + "idx": 1163, + "ok": false + }, + { + "idx": 1287, + "ok": false + }, + { + "idx": 282, + "ok": false + }, + { + "idx": 321, + "ok": true + }, + { + "idx": 252, + "ok": false + }, + { + "idx": 246, + "ok": true + }, + { + "idx": 781, + "ok": false + }, + { + "idx": 821, + "ok": true + }, + { + "idx": 1210, + "ok": true + }, + { + "idx": 958, + "ok": false + }, + { + "idx": 285, + "ok": true + }, + { + "idx": 1146, + "ok": true + }, + { + "idx": 612, + "ok": true + }, + { + "idx": 725, + "ok": true + }, + { + "idx": 1294, + "ok": false + }, + { + "idx": 968, + "ok": true + }, + { + "idx": 850, + "ok": false + }, + { + "idx": 446, + "ok": false + }, + { + "idx": 976, + "ok": false + }, + { + "idx": 1001, + "ok": false + }, + { + "idx": 1027, + "ok": true + }, + { + "idx": 651, + "ok": true + }, + { + "idx": 1009, + "ok": false + }, + { + "idx": 123, + "ok": true + }, + { + "idx": 909, + "ok": true + }, + { + "idx": 614, + "ok": false + }, + { + "idx": 292, + "ok": true + }, + { + "idx": 1014, + "ok": false + }, + { + "idx": 107, + "ok": false + }, + { + "idx": 1272, + "ok": false + }, + { + "idx": 441, + "ok": true + }, + { + "idx": 52, + "ok": true + }, + { + "idx": 727, + "ok": true + }, + { + "idx": 965, + "ok": true + }, + { + "idx": 800, + "ok": true + }, + { + "idx": 20, + "ok": false + }, + { + "idx": 1078, + "ok": true + }, + { + "idx": 136, + "ok": true + }, + { + "idx": 166, + "ok": true + }, + { + "idx": 809, + "ok": false + }, + { + "idx": 13, + "ok": false + }, + { + "idx": 739, + "ok": true + }, + { + "idx": 84, + "ok": true + }, + { + "idx": 237, + "ok": true + }, + { + "idx": 1271, + "ok": true + }, + { + "idx": 7, + "ok": false + }, + { + "idx": 553, + "ok": true + }, + { + "idx": 1311, + "ok": true + }, + { + "idx": 599, + "ok": true + }, + { + "idx": 464, + "ok": false + }, + { + "idx": 288, + "ok": false + }, + { + "idx": 1173, + "ok": false + }, + { + "idx": 589, + "ok": false + }, + { + "idx": 391, + "ok": true + }, + { + "idx": 888, + "ok": true + }, + { + "idx": 943, + "ok": false + }, + { + "idx": 676, + "ok": false + }, + { + "idx": 786, + "ok": true + }, + { + "idx": 344, + "ok": true + }, + { + "idx": 677, + "ok": true + }, + { + "idx": 863, + "ok": false + }, + { + "idx": 891, + "ok": true + }, + { + "idx": 303, + "ok": false + }, + { + "idx": 916, + "ok": false + }, + { + "idx": 302, + "ok": true + }, + { + "idx": 1073, + "ok": false + }, + { + "idx": 647, + "ok": false + }, + { + "idx": 264, + "ok": true + }, + { + "idx": 427, + "ok": false + }, + { + "idx": 382, + "ok": false + }, + { + "idx": 715, + "ok": true + }, + { + "idx": 796, + "ok": false + }, + { + "idx": 875, + "ok": true + }, + { + "idx": 1007, + "ok": false + }, + { + "idx": 797, + "ok": true + }, + { + "idx": 450, + "ok": false + }, + { + "idx": 401, + "ok": true + }, + { + "idx": 899, + "ok": true + }, + { + "idx": 418, + "ok": false + }, + { + "idx": 1201, + "ok": true + }, + { + "idx": 101, + "ok": false + }, + { + "idx": 795, + "ok": true + }, + { + "idx": 68, + "ok": true + }, + { + "idx": 479, + "ok": false + }, + { + "idx": 1296, + "ok": false + }, + { + "idx": 173, + "ok": false + }, + { + "idx": 744, + "ok": true + }, + { + "idx": 116, + "ok": true + }, + { + "idx": 1306, + "ok": false + }, + { + "idx": 355, + "ok": false + }, + { + "idx": 477, + "ok": true + }, + { + "idx": 1250, + "ok": true + }, + { + "idx": 609, + "ok": true + }, + { + "idx": 1254, + "ok": true + }, + { + "idx": 177, + "ok": false + }, + { + "idx": 1048, + "ok": false + }, + { + "idx": 582, + "ok": false + }, + { + "idx": 723, + "ok": true + }, + { + "idx": 842, + "ok": true + }, + { + "idx": 938, + "ok": true + }, + { + "idx": 110, + "ok": true + }, + { + "idx": 1057, + "ok": false + }, + { + "idx": 1120, + "ok": false + }, + { + "idx": 881, + "ok": true + }, + { + "idx": 1189, + "ok": false + }, + { + "idx": 931, + "ok": true + }, + { + "idx": 1003, + "ok": false + }, + { + "idx": 521, + "ok": false + }, + { + "idx": 974, + "ok": true + }, + { + "idx": 690, + "ok": false + }, + { + "idx": 544, + "ok": false + }, + { + "idx": 86, + "ok": false + }, + { + "idx": 89, + "ok": false + }, + { + "idx": 333, + "ok": true + }, + { + "idx": 716, + "ok": true + }, + { + "idx": 14, + "ok": false + }, + { + "idx": 287, + "ok": true + }, + { + "idx": 130, + "ok": false + }, + { + "idx": 876, + "ok": false + }, + { + "idx": 454, + "ok": false + }, + { + "idx": 1246, + "ok": true + }, + { + "idx": 812, + "ok": false + }, + { + "idx": 1142, + "ok": true + }, + { + "idx": 452, + "ok": true + }, + { + "idx": 929, + "ok": true + }, + { + "idx": 394, + "ok": false + }, + { + "idx": 695, + "ok": true + }, + { + "idx": 1245, + "ok": true + }, + { + "idx": 209, + "ok": false + }, + { + "idx": 1241, + "ok": false + }, + { + "idx": 175, + "ok": false + }, + { + "idx": 653, + "ok": false + }, + { + "idx": 661, + "ok": true + }, + { + "idx": 1097, + "ok": false + }, + { + "idx": 933, + "ok": false + }, + { + "idx": 665, + "ok": true + }, + { + "idx": 523, + "ok": false + }, + { + "idx": 59, + "ok": true + }, + { + "idx": 1069, + "ok": true + }, + { + "idx": 90, + "ok": false + }, + { + "idx": 389, + "ok": false + }, + { + "idx": 755, + "ok": true + }, + { + "idx": 164, + "ok": true + }, + { + "idx": 429, + "ok": true + }, + { + "idx": 1074, + "ok": true + }, + { + "idx": 708, + "ok": true + }, + { + "idx": 384, + "ok": false + }, + { + "idx": 412, + "ok": true + }, + { + "idx": 514, + "ok": true + }, + { + "idx": 617, + "ok": false + }, + { + "idx": 639, + "ok": true + }, + { + "idx": 1058, + "ok": false + }, + { + "idx": 787, + "ok": true + }, + { + "idx": 987, + "ok": true + }, + { + "idx": 704, + "ok": true + }, + { + "idx": 491, + "ok": true + }, + { + "idx": 91, + "ok": true + }, + { + "idx": 626, + "ok": false + }, + { + "idx": 1130, + "ok": false + }, + { + "idx": 148, + "ok": false + }, + { + "idx": 18, + "ok": false + }, + { + "idx": 897, + "ok": false + }, + { + "idx": 97, + "ok": false + }, + { + "idx": 844, + "ok": false + }, + { + "idx": 1010, + "ok": true + }, + { + "idx": 901, + "ok": false + }, + { + "idx": 241, + "ok": false + }, + { + "idx": 493, + "ok": false + }, + { + "idx": 202, + "ok": true + }, + { + "idx": 314, + "ok": false + }, + { + "idx": 847, + "ok": false + }, + { + "idx": 437, + "ok": true + }, + { + "idx": 902, + "ok": true + }, + { + "idx": 157, + "ok": false + }, + { + "idx": 874, + "ok": true + }, + { + "idx": 1144, + "ok": false + }, + { + "idx": 807, + "ok": false + }, + { + "idx": 80, + "ok": false + }, + { + "idx": 369, + "ok": false + }, + { + "idx": 511, + "ok": false + }, + { + "idx": 262, + "ok": true + }, + { + "idx": 571, + "ok": true + }, + { + "idx": 721, + "ok": false + }, + { + "idx": 654, + "ok": true + }, + { + "idx": 890, + "ok": true + }, + { + "idx": 219, + "ok": false + }, + { + "idx": 1140, + "ok": false + }, + { + "idx": 584, + "ok": false + }, + { + "idx": 1249, + "ok": false + }, + { + "idx": 1108, + "ok": true + }, + { + "idx": 607, + "ok": false + }, + { + "idx": 904, + "ok": true + }, + { + "idx": 1052, + "ok": true + }, + { + "idx": 1238, + "ok": true + }, + { + "idx": 946, + "ok": false + }, + { + "idx": 1098, + "ok": true + }, + { + "idx": 534, + "ok": false + }, + { + "idx": 558, + "ok": true + }, + { + "idx": 474, + "ok": true + }, + { + "idx": 33, + "ok": true + }, + { + "idx": 243, + "ok": false + }, + { + "idx": 1259, + "ok": false + }, + { + "idx": 353, + "ok": false + }, + { + "idx": 509, + "ok": false + }, + { + "idx": 1096, + "ok": false + }, + { + "idx": 1055, + "ok": true + }, + { + "idx": 877, + "ok": true + }, + { + "idx": 249, + "ok": true + }, + { + "idx": 1156, + "ok": false + }, + { + "idx": 735, + "ok": true + }, + { + "idx": 470, + "ok": true + }, + { + "idx": 1119, + "ok": false + }, + { + "idx": 577, + "ok": false + }, + { + "idx": 453, + "ok": false + }, + { + "idx": 132, + "ok": false + }, + { + "idx": 1061, + "ok": true + }, + { + "idx": 629, + "ok": true + }, + { + "idx": 670, + "ok": true + }, + { + "idx": 478, + "ok": false + }, + { + "idx": 764, + "ok": true + }, + { + "idx": 983, + "ok": true + }, + { + "idx": 587, + "ok": false + }, + { + "idx": 1192, + "ok": false + }, + { + "idx": 350, + "ok": false + }, + { + "idx": 281, + "ok": false + }, + { + "idx": 31, + "ok": false + }, + { + "idx": 1132, + "ok": false + }, + { + "idx": 1033, + "ok": true + }, + { + "idx": 671, + "ok": true + }, + { + "idx": 751, + "ok": true + } + ] + }, + "2:6": { + "acc": 0.546875, + "by_label": { + "drop": 0.31, + "easy": 0.8275862068965517, + "hard": 0.6692913385826772 + }, + "per_item": [ + { + "idx": 1092, + "ok": false + }, + { + "idx": 696, + "ok": true + }, + { + "idx": 242, + "ok": false + }, + { + "idx": 1085, + "ok": true + }, + { + "idx": 510, + "ok": false + }, + { + "idx": 334, + "ok": true + }, + { + "idx": 139, + "ok": false + }, + { + "idx": 849, + "ok": true + }, + { + "idx": 593, + "ok": true + }, + { + "idx": 579, + "ok": true + }, + { + "idx": 1064, + "ok": false + }, + { + "idx": 275, + "ok": false + }, + { + "idx": 1175, + "ok": false + }, + { + "idx": 1071, + "ok": false + }, + { + "idx": 1284, + "ok": false + }, + { + "idx": 430, + "ok": false + }, + { + "idx": 1088, + "ok": false + }, + { + "idx": 215, + "ok": true + }, + { + "idx": 841, + "ok": false + }, + { + "idx": 1299, + "ok": true + }, + { + "idx": 1113, + "ok": true + }, + { + "idx": 826, + "ok": false + }, + { + "idx": 570, + "ok": true + }, + { + "idx": 598, + "ok": true + }, + { + "idx": 905, + "ok": false + }, + { + "idx": 761, + "ok": true + }, + { + "idx": 1163, + "ok": false + }, + { + "idx": 1287, + "ok": false + }, + { + "idx": 282, + "ok": false + }, + { + "idx": 321, + "ok": true + }, + { + "idx": 252, + "ok": false + }, + { + "idx": 246, + "ok": true + }, + { + "idx": 781, + "ok": false + }, + { + "idx": 821, + "ok": true + }, + { + "idx": 1210, + "ok": true + }, + { + "idx": 958, + "ok": false + }, + { + "idx": 285, + "ok": true + }, + { + "idx": 1146, + "ok": true + }, + { + "idx": 612, + "ok": true + }, + { + "idx": 725, + "ok": true + }, + { + "idx": 1294, + "ok": false + }, + { + "idx": 968, + "ok": false + }, + { + "idx": 850, + "ok": true + }, + { + "idx": 446, + "ok": false + }, + { + "idx": 976, + "ok": false + }, + { + "idx": 1001, + "ok": false + }, + { + "idx": 1027, + "ok": true + }, + { + "idx": 651, + "ok": true + }, + { + "idx": 1009, + "ok": false + }, + { + "idx": 123, + "ok": true + }, + { + "idx": 909, + "ok": true + }, + { + "idx": 614, + "ok": false + }, + { + "idx": 292, + "ok": true + }, + { + "idx": 1014, + "ok": false + }, + { + "idx": 107, + "ok": false + }, + { + "idx": 1272, + "ok": false + }, + { + "idx": 441, + "ok": true + }, + { + "idx": 52, + "ok": true + }, + { + "idx": 727, + "ok": true + }, + { + "idx": 965, + "ok": true + }, + { + "idx": 800, + "ok": true + }, + { + "idx": 20, + "ok": false + }, + { + "idx": 1078, + "ok": true + }, + { + "idx": 136, + "ok": true + }, + { + "idx": 166, + "ok": true + }, + { + "idx": 809, + "ok": false + }, + { + "idx": 13, + "ok": false + }, + { + "idx": 739, + "ok": true + }, + { + "idx": 84, + "ok": true + }, + { + "idx": 237, + "ok": true + }, + { + "idx": 1271, + "ok": true + }, + { + "idx": 7, + "ok": false + }, + { + "idx": 553, + "ok": true + }, + { + "idx": 1311, + "ok": true + }, + { + "idx": 599, + "ok": true + }, + { + "idx": 464, + "ok": false + }, + { + "idx": 288, + "ok": true + }, + { + "idx": 1173, + "ok": false + }, + { + "idx": 589, + "ok": false + }, + { + "idx": 391, + "ok": true + }, + { + "idx": 888, + "ok": true + }, + { + "idx": 943, + "ok": false + }, + { + "idx": 676, + "ok": true + }, + { + "idx": 786, + "ok": true + }, + { + "idx": 344, + "ok": true + }, + { + "idx": 677, + "ok": true + }, + { + "idx": 863, + "ok": true + }, + { + "idx": 891, + "ok": true + }, + { + "idx": 303, + "ok": false + }, + { + "idx": 916, + "ok": false + }, + { + "idx": 302, + "ok": true + }, + { + "idx": 1073, + "ok": false + }, + { + "idx": 647, + "ok": false + }, + { + "idx": 264, + "ok": true + }, + { + "idx": 427, + "ok": false + }, + { + "idx": 382, + "ok": false + }, + { + "idx": 715, + "ok": true + }, + { + "idx": 796, + "ok": false + }, + { + "idx": 875, + "ok": true + }, + { + "idx": 1007, + "ok": false + }, + { + "idx": 797, + "ok": true + }, + { + "idx": 450, + "ok": false + }, + { + "idx": 401, + "ok": false + }, + { + "idx": 899, + "ok": true + }, + { + "idx": 418, + "ok": false + }, + { + "idx": 1201, + "ok": true + }, + { + "idx": 101, + "ok": false + }, + { + "idx": 795, + "ok": true + }, + { + "idx": 68, + "ok": false + }, + { + "idx": 479, + "ok": true + }, + { + "idx": 1296, + "ok": true + }, + { + "idx": 173, + "ok": false + }, + { + "idx": 744, + "ok": true + }, + { + "idx": 116, + "ok": true + }, + { + "idx": 1306, + "ok": false + }, + { + "idx": 355, + "ok": false + }, + { + "idx": 477, + "ok": true + }, + { + "idx": 1250, + "ok": true + }, + { + "idx": 609, + "ok": true + }, + { + "idx": 1254, + "ok": true + }, + { + "idx": 177, + "ok": false + }, + { + "idx": 1048, + "ok": false + }, + { + "idx": 582, + "ok": false + }, + { + "idx": 723, + "ok": true + }, + { + "idx": 842, + "ok": true + }, + { + "idx": 938, + "ok": true + }, + { + "idx": 110, + "ok": true + }, + { + "idx": 1057, + "ok": false + }, + { + "idx": 1120, + "ok": true + }, + { + "idx": 881, + "ok": true + }, + { + "idx": 1189, + "ok": false + }, + { + "idx": 931, + "ok": false + }, + { + "idx": 1003, + "ok": false + }, + { + "idx": 521, + "ok": false + }, + { + "idx": 974, + "ok": true + }, + { + "idx": 690, + "ok": false + }, + { + "idx": 544, + "ok": true + }, + { + "idx": 86, + "ok": false + }, + { + "idx": 89, + "ok": false + }, + { + "idx": 333, + "ok": true + }, + { + "idx": 716, + "ok": true + }, + { + "idx": 14, + "ok": true + }, + { + "idx": 287, + "ok": true + }, + { + "idx": 130, + "ok": false + }, + { + "idx": 876, + "ok": false + }, + { + "idx": 454, + "ok": false + }, + { + "idx": 1246, + "ok": true + }, + { + "idx": 812, + "ok": false + }, + { + "idx": 1142, + "ok": true + }, + { + "idx": 452, + "ok": false + }, + { + "idx": 929, + "ok": true + }, + { + "idx": 394, + "ok": false + }, + { + "idx": 695, + "ok": true + }, + { + "idx": 1245, + "ok": true + }, + { + "idx": 209, + "ok": false + }, + { + "idx": 1241, + "ok": false + }, + { + "idx": 175, + "ok": false + }, + { + "idx": 653, + "ok": false + }, + { + "idx": 661, + "ok": true + }, + { + "idx": 1097, + "ok": true + }, + { + "idx": 933, + "ok": false + }, + { + "idx": 665, + "ok": true + }, + { + "idx": 523, + "ok": false + }, + { + "idx": 59, + "ok": true + }, + { + "idx": 1069, + "ok": true + }, + { + "idx": 90, + "ok": true + }, + { + "idx": 389, + "ok": false + }, + { + "idx": 755, + "ok": true + }, + { + "idx": 164, + "ok": true + }, + { + "idx": 429, + "ok": true + }, + { + "idx": 1074, + "ok": true + }, + { + "idx": 708, + "ok": true + }, + { + "idx": 384, + "ok": false + }, + { + "idx": 412, + "ok": true + }, + { + "idx": 514, + "ok": true + }, + { + "idx": 617, + "ok": false + }, + { + "idx": 639, + "ok": true + }, + { + "idx": 1058, + "ok": false + }, + { + "idx": 787, + "ok": true + }, + { + "idx": 987, + "ok": true + }, + { + "idx": 704, + "ok": true + }, + { + "idx": 491, + "ok": true + }, + { + "idx": 91, + "ok": false + }, + { + "idx": 626, + "ok": true + }, + { + "idx": 1130, + "ok": false + }, + { + "idx": 148, + "ok": true + }, + { + "idx": 18, + "ok": false + }, + { + "idx": 897, + "ok": true + }, + { + "idx": 97, + "ok": false + }, + { + "idx": 844, + "ok": false + }, + { + "idx": 1010, + "ok": true + }, + { + "idx": 901, + "ok": false + }, + { + "idx": 241, + "ok": false + }, + { + "idx": 493, + "ok": false + }, + { + "idx": 202, + "ok": true + }, + { + "idx": 314, + "ok": true + }, + { + "idx": 847, + "ok": true + }, + { + "idx": 437, + "ok": true + }, + { + "idx": 902, + "ok": true + }, + { + "idx": 157, + "ok": false + }, + { + "idx": 874, + "ok": true + }, + { + "idx": 1144, + "ok": false + }, + { + "idx": 807, + "ok": true + }, + { + "idx": 80, + "ok": false + }, + { + "idx": 369, + "ok": false + }, + { + "idx": 511, + "ok": true + }, + { + "idx": 262, + "ok": true + }, + { + "idx": 571, + "ok": true + }, + { + "idx": 721, + "ok": false + }, + { + "idx": 654, + "ok": true + }, + { + "idx": 890, + "ok": true + }, + { + "idx": 219, + "ok": false + }, + { + "idx": 1140, + "ok": false + }, + { + "idx": 584, + "ok": false + }, + { + "idx": 1249, + "ok": false + }, + { + "idx": 1108, + "ok": true + }, + { + "idx": 607, + "ok": false + }, + { + "idx": 904, + "ok": true + }, + { + "idx": 1052, + "ok": true + }, + { + "idx": 1238, + "ok": true + }, + { + "idx": 946, + "ok": false + }, + { + "idx": 1098, + "ok": true + }, + { + "idx": 534, + "ok": false + }, + { + "idx": 558, + "ok": true + }, + { + "idx": 474, + "ok": true + }, + { + "idx": 33, + "ok": false + }, + { + "idx": 243, + "ok": true + }, + { + "idx": 1259, + "ok": true + }, + { + "idx": 353, + "ok": false + }, + { + "idx": 509, + "ok": false + }, + { + "idx": 1096, + "ok": false + }, + { + "idx": 1055, + "ok": true + }, + { + "idx": 877, + "ok": true + }, + { + "idx": 249, + "ok": true + }, + { + "idx": 1156, + "ok": false + }, + { + "idx": 735, + "ok": true + }, + { + "idx": 470, + "ok": true + }, + { + "idx": 1119, + "ok": false + }, + { + "idx": 577, + "ok": false + }, + { + "idx": 453, + "ok": false + }, + { + "idx": 132, + "ok": false + }, + { + "idx": 1061, + "ok": true + }, + { + "idx": 629, + "ok": true + }, + { + "idx": 670, + "ok": false + }, + { + "idx": 478, + "ok": false + }, + { + "idx": 764, + "ok": true + }, + { + "idx": 983, + "ok": true + }, + { + "idx": 587, + "ok": false + }, + { + "idx": 1192, + "ok": false + }, + { + "idx": 350, + "ok": true + }, + { + "idx": 281, + "ok": false + }, + { + "idx": 31, + "ok": true + }, + { + "idx": 1132, + "ok": false + }, + { + "idx": 1033, + "ok": true + }, + { + "idx": 671, + "ok": true + }, + { + "idx": 751, + "ok": true + } + ] + } + }, + "n": 256 +} \ No newline at end of file diff --git a/results-loop/gsm_cot_data.json b/results-loop/gsm_cot_data.json new file mode 100644 index 0000000..c8174d1 --- /dev/null +++ b/results-loop/gsm_cot_data.json @@ -0,0 +1,2137 @@ +[ + { + "idx": 7292, + "label": "hard", + "cot": "5*10=50\n2*20=40 (Assuming the online books cost $20 each, since $40/2=20)\n3*40=120\n50+40+120=210\nAnswer: 210" + }, + { + "idx": 4188, + "label": "easy", + "cot": "80*0.15=12\n80-12=68\nAnswer: 68" + }, + { + "idx": 3980, + "label": "hard", + "cot": "2*2=4\nBluebirds = 4\nCardinals = 3*4=12\nTotal birds = 12+4+2=18\nAnswer: 18" + }, + { + "idx": 3317, + "label": "hard", + "cot": "180*0.05=9\n180-6=174\n9-6=3\n\nAnswer: 3" + }, + { + "idx": 2933, + "label": "hard", + "cot": "235-227=8\n8+56=64\n64-35=29\n\nAnswer: 29" + }, + { + "idx": 2308, + "label": "easy", + "cot": "1. $24 - 8 = 16$\n2. $16 - 2 - 3 = 11$\n3. Answer: 11" + }, + { + "idx": 6191, + "label": "hard", + "cot": "60000*0.75=45000\n45000-5000=40000\nAnswer: 40000" + }, + { + "idx": 776, + "label": "hard", + "cot": "Amber worked 12 hours.\nArmand worked $12/3=4$ hours and Ella worked $2*12=24$ hours.\n$12+4+24=40$\nAnswer: 40" + }, + { + "idx": 5065, + "label": "hard", + "cot": "Martha = 2\nJenna = 4*2-2 = 6\nMark = 6/2 = 3\nTotal finished = 2+6+3 = 11\nUnfinished = 20-11 = 9\nAnswer: 9" + }, + { + "idx": 4362, + "label": "hard", + "cot": "40*0.75=30\n50*0.98=49\n30+49=79\nAnswer: 79" + }, + { + "idx": 7390, + "label": "hard", + "cot": "Let $A$ be your current age.\n$A = 3(A-6)$\n$A = 3A - 18$\n$18 = 2A$\n$A = 9$\n\nAnswer: 9" + }, + { + "idx": 2540, + "label": "easy", + "cot": "4000*3=12000\n12000-4000=8000\nAnswer: 8000" + }, + { + "idx": 809, + "label": "easy", + "cot": "1. Calculate daily miles: $1.5 + 1.5 = 3$ miles.\n2. Calculate total miles: $3 \\times 30 = 90$ miles.\n3. Final result is 90.\n\nAnswer: 90" + }, + { + "idx": 6967, + "label": "hard", + "cot": "R=20\nA=2*20+4=44\nTotal=20+44=64\nAnswer: 64" + }, + { + "idx": 5603, + "label": "hard", + "cot": "150*1.3=195\n195+10=205\nAnswer: 205" + }, + { + "idx": 2704, + "label": "easy", + "cot": "4000*0.30=1200\n4000-1200=2800\nAnswer: 2800" + }, + { + "idx": 4585, + "label": "easy", + "cot": "2000 / 200 = 10\nAnswer: 10" + }, + { + "idx": 824, + "label": "hard", + "cot": "15/3=5, 15-5=10, 5*30+10*25=150+250=400\nAnswer: 400" + }, + { + "idx": 5004, + "label": "easy", + "cot": "4*20=80\n80*5=400\nAnswer: 400" + }, + { + "idx": 1675, + "label": "hard", + "cot": "500+20*60=1700\n20*60=1200\n500+1200=1700\nAnswer: 1700" + }, + { + "idx": 7088, + "label": "easy", + "cot": "2.00+0.50*10=7.00\nAnswer: 7.00" + }, + { + "idx": 510, + "label": "easy", + "cot": "3*10=30\n30*6=180\nAnswer: 180" + }, + { + "idx": 6594, + "label": "easy", + "cot": "65-22-24=19\n65-22-24=19\nAnswer: 19" + }, + { + "idx": 764, + "label": "hard", + "cot": "1. $40 * 3/4 = 30$\n2. $30 - 25 = 5$\n3. Answer: 5" + }, + { + "idx": 5895, + "label": "hard", + "cot": "Mark's jump: 6\nLisa's jump: 6*2=12\nJacob's jump: 12*2=24\nJames's jump: 24*(2/3)=16\n\nAnswer: 16" + }, + { + "idx": 5818, + "label": "hard", + "cot": "P=40\nD=50\nN=100\nAnswer: 100" + }, + { + "idx": 5122, + "label": "hard", + "cot": "1. $5 \\times 20 = 100$\n2. $4000 \\div 100 = 40$\n3. $40$\nAnswer: 40" + }, + { + "idx": 5012, + "label": "hard", + "cot": "1. Button-up shirts = 3\n2. Sweater vests = 2 * 3 = 6\n3. Outfits = 3 * 6 = 18\n\nAnswer: 18" + }, + { + "idx": 4043, + "label": "hard", + "cot": "1000\n1000*1.5=1500\n1000+1500+1500=4000\nAnswer: 4000" + }, + { + "idx": 2729, + "label": "easy", + "cot": "1. Total man-hours needed: $3 \\text{ men/hour} \\times 5 \\text{ jobs} \\times 1 \\text{ hour/job} = 15$ man-hours.\n2. Total earnings: $15 \\text{ hours} \\times \\$10/\\text{hour} = \\$150$.\n3. Total money earned: $150$.\n\nAnswer: 150" + }, + { + "idx": 1998, + "label": "easy", + "cot": "100-20=80\n80/5=16\nAnswer: 16" + }, + { + "idx": 5982, + "label": "hard", + "cot": "4*5=20\n6*2=12\n20+12=32\nAnswer: 32" + }, + { + "idx": 2664, + "label": "easy", + "cot": "4*12=48\n48-39=9\nAnswer: 9" + }, + { + "idx": 5764, + "label": "easy", + "cot": "2*4=8\n8+4=12\nAnswer: 12" + }, + { + "idx": 1565, + "label": "hard", + "cot": "2 + 3 + 3 + 5 = 13\nAnswer: 13" + }, + { + "idx": 1954, + "label": "hard", + "cot": "12\n12*6=72\n12+72=84\nAnswer: 84" + }, + { + "idx": 6579, + "label": "easy", + "cot": "100*0.8=80\n100-80=20\nAnswer: 20" + }, + { + "idx": 747, + "label": "easy", + "cot": "500\n1000\n1500\nAnswer: 1500" + }, + { + "idx": 7168, + "label": "hard", + "cot": "60*1/4=15\n15*0.20=3\nAnswer: 3" + }, + { + "idx": 4161, + "label": "hard", + "cot": "50*400=20000\n20000*0.60=12000\n20000-12000=8000\nAnswer: 8000" + }, + { + "idx": 2469, + "label": "hard", + "cot": "8+6=14\n6*2/3=4\n24-14-4=6\nAnswer: 6" + }, + { + "idx": 4484, + "label": "hard", + "cot": "5000-4600=400\n400*12=4800\nAnswer: 4800" + }, + { + "idx": 2725, + "label": "hard", + "cot": "40*0.6=24\n24*(3/4)*10=180\n180-130=50\nAnswer: 50" + }, + { + "idx": 1664, + "label": "hard", + "cot": "6*2=12\n4/2=2\n12+2=14\nAnswer: 14" + }, + { + "idx": 4941, + "label": "hard", + "cot": "10*20=200\n20*30=600\n200+600=800\nAnswer: 800" + }, + { + "idx": 4482, + "label": "hard", + "cot": "18 streets means 9 trees are fruit trees.\nDivide 9 by 3 for each fruit tree type: 9/3=3.\nAnswer: 3" + }, + { + "idx": 4813, + "label": "easy", + "cot": "3*40=120\n3*60=180\n120+180=300\nAnswer: 300" + }, + { + "idx": 750, + "label": "hard", + "cot": "4 * 6 = 24\n24 * 2 = 48\nAnswer: 48" + }, + { + "idx": 4884, + "label": "easy", + "cot": "200*0.25=50\n200-50=150\nAnswer: 150" + }, + { + "idx": 4715, + "label": "hard", + "cot": "1. Weekly cost: $(2 \\times 10) + (1 \\times 5) = 25$\n2. Total bill before discount/fine: $25 \\times 4 = 100$\n3. Final bill: $(100 \\times 0.82) + 20 = 82 + 20 = 102$\n\nAnswer: 102" + }, + { + "idx": 2378, + "label": "hard", + "cot": "250+120+35=405\n3*250=750\n7*120=840\n10*35=350\n750+840+350=1940\nAnswer: 1940" + }, + { + "idx": 6729, + "label": "easy", + "cot": "1. $36 \\times 2 = 72$\n2. $72 \\times 1.5 = 108$\n3. Answer: 108" + }, + { + "idx": 2130, + "label": "hard", + "cot": "1. Second course cost is $15+5=20$.\n2. Dessert cost is $0.25 \\times 20 = 5$.\n3. Total cost is $15+20+5=40$, and remaining money is $60-40=20$.\nAnswer: 20" + }, + { + "idx": 735, + "label": "easy", + "cot": "1. Convert minutes to hours: $10 \\text{ minutes} / 60 \\text{ minutes/hour} = 1/6 \\text{ hours}$.\n2. Calculate speed: $5 \\text{ miles} / (1/6 \\text{ hours}) = 5 * 6 = 30 \\text{ mph}$.\n3. Final result is 30.\n\nAnswer: 30" + }, + { + "idx": 5560, + "label": "hard", + "cot": "108/2=54\n54-6=48\n48/4=12\n\nAnswer: 12" + }, + { + "idx": 316, + "label": "easy", + "cot": "220*0.15=33\n220+33=253\nAnswer: 253" + }, + { + "idx": 7357, + "label": "hard", + "cot": "Area of blue sand: $6*7=42$\nArea of red sand: $5*5=25$\nTotal area: $42+25=67$\nTotal sand needed: $67*3=201$\n\nAnswer: 201" + }, + { + "idx": 5728, + "label": "easy", + "cot": "1. Cost of one slice of cake is $2.40 \\times 3/4 = 1.80$.\n2. Cost of 2 slices of cake is $2 \\times 1.80 = 3.60$.\n3. Total cost is $3.60 + 2.40 = 6.00$.\nAnswer: 6.00" + }, + { + "idx": 6863, + "label": "hard", + "cot": "500/5=100\n100/4=25\nAnswer: 25" + }, + { + "idx": 4297, + "label": "easy", + "cot": "10*2+5*5=45\n10+5=15\n45/15=3\n\nAnswer: 3" + }, + { + "idx": 2257, + "label": "hard", + "cot": "4000*3/4=3000\n3000*1/4=750\nAnswer: 750" + }, + { + "idx": 1929, + "label": "hard", + "cot": "Sydney = 300 / 15\nLori = 300\nTotal = 300 + (300 / 15)\n\nAnswer: 320" + }, + { + "idx": 7332, + "label": "easy", + "cot": "1. $45000 / 3 = 15000$\n2. $15000 * 1.5 = 22500$\n3. $45000 + 22500 = 67500$\nAnswer: 67500" + }, + { + "idx": 4831, + "label": "easy", + "cot": "2*10=20\n20-1=19\nAnswer: 19" + }, + { + "idx": 4748, + "label": "hard", + "cot": "6+3=9\n9+15=24\n24" + }, + { + "idx": 2254, + "label": "hard", + "cot": "52 + 2 = 54\n54 / 3 = 18\nAnswer: 18" + }, + { + "idx": 4035, + "label": "easy", + "cot": "10*2=20\n20*7=140\nAnswer: 140" + }, + { + "idx": 5736, + "label": "hard", + "cot": "2*14=28\n28-6=22\nAnswer: 22" + }, + { + "idx": 2656, + "label": "easy", + "cot": "W=200\nX=100\nY=300\nZ=400\n200+100+300+400=1000\nAnswer: 1000" + }, + { + "idx": 944, + "label": "hard", + "cot": "5+9=14\n14/2=7\nAnswer: 7" + }, + { + "idx": 3984, + "label": "hard", + "cot": "Judson = 500\nKenny = 500 * 1.20 = 600\nCamilo = 600 + 200 = 800\nTotal cost = 500 + 600 + 800 = 1900\n\nAnswer: 1900" + }, + { + "idx": 5162, + "label": "easy", + "cot": "360/3=120\n240/120=2\nAnswer: 2" + }, + { + "idx": 1990, + "label": "hard", + "cot": "1970 + 10 = 1980\n1980 + 20 = 2000\nAnswer: 2000" + }, + { + "idx": 132, + "label": "easy", + "cot": "5*4=20\n20*5.50=110\nAnswer: 110.0" + }, + { + "idx": 959, + "label": "easy", + "cot": "50+150=200\n50/200=0.25\n0.25*100=25\nAnswer: 25" + }, + { + "idx": 1806, + "label": "easy", + "cot": "5*2=10\n5+10=15\nAnswer: 15" + }, + { + "idx": 3047, + "label": "hard", + "cot": "20*10=200\n200*3/4=150\n150*10=1500\nAnswer: 1500" + }, + { + "idx": 509, + "label": "easy", + "cot": "$0.10 * 2.4 = 0.24$\n$0.24 * 25 = 6$\nAnswer: 6" + }, + { + "idx": 1198, + "label": "hard", + "cot": "200*4/5=160\n200-160=40\nAnswer: 40" + }, + { + "idx": 5715, + "label": "hard", + "cot": "-14 + (-8) + 1 = -21\n-21 / 3 = -7\nAnswer: -7" + }, + { + "idx": 5195, + "label": "hard", + "cot": "5*2=10\n1+2+3+3=9\n10-9=1\n\nAnswer: 1" + }, + { + "idx": 7454, + "label": "hard", + "cot": "20*2*30=1200\n1200/60=20\nAnswer: 20" + }, + { + "idx": 6800, + "label": "easy", + "cot": "20 - 7 - 3 + 4 + 2 = 16\nAnswer: 16" + }, + { + "idx": 4718, + "label": "hard", + "cot": "5*3=15\n735/15=49\n49/7=7\n\nAnswer: 7" + }, + { + "idx": 3204, + "label": "easy", + "cot": "24 eggs / 4 people = 6\nAnswer: 6" + }, + { + "idx": 749, + "label": "easy", + "cot": "4*4=16\n16-4=12\nAnswer: 12" + }, + { + "idx": 950, + "label": "hard", + "cot": "10*20+20*5=300\n300*0.10=30\n300-30=270\nAnswer: 270" + }, + { + "idx": 4960, + "label": "hard", + "cot": "10 + 13 + 17 = 40\n40 * 0.10 = 4\nAnswer: 4" + }, + { + "idx": 1594, + "label": "easy", + "cot": "1*20=20\n2*20=40\n20+40=60\n90-60=30\nAnswer: 30" + }, + { + "idx": 1515, + "label": "hard", + "cot": "145+5=150\n150+9=159\nAnswer: 159" + }, + { + "idx": 1014, + "label": "easy", + "cot": "2*2=4\n2+4=6\nAnswer: 6" + }, + { + "idx": 3925, + "label": "easy", + "cot": "10+2=12\n20-12=8\nAnswer: 8" + }, + { + "idx": 1725, + "label": "hard", + "cot": "1. MRI cost is $250 * 3 = 750$.\n2. Total cost is $250 + 750 = 1000$.\n3. Mike paid $1000 * (1 - 0.80) = 200$.\n\nAnswer: 200" + }, + { + "idx": 500, + "label": "hard", + "cot": "23*2=46\n46-4=42\n42/3+16=30" + }, + { + "idx": 5565, + "label": "easy", + "cot": "120/2=60\n60*0.9=54\nAnswer: 54" + }, + { + "idx": 573, + "label": "hard", + "cot": "8*18=144\n7*20=140\n144-140=4\nAnswer: 4" + }, + { + "idx": 1809, + "label": "hard", + "cot": "91+90+92=273\n273/3=91\nAnswer: 91" + }, + { + "idx": 2466, + "label": "hard", + "cot": "380-57=323\n323-24=299\nAnswer: 299" + }, + { + "idx": 2869, + "label": "hard", + "cot": "120*0.25=30\n30+10=40\n120-40=80\nAnswer: 80" + }, + { + "idx": 3572, + "label": "hard", + "cot": "1. Blouses ironed: $120/15 = 8$\n2. Dresses ironed: $180/20 = 9$\n3. Total pieces: $8+9 = 17$\n\nAnswer: 17" + }, + { + "idx": 3826, + "label": "hard", + "cot": "S = shirt cost, H = shoe cost\nS + H = 300\nH = 2S + 9\nS + (2S + 9) = 300\n3S + 9 = 300\n3S = 291\nS = 291 / 3\nS = 97\n\nAnswer: 97" + }, + { + "idx": 826, + "label": "easy", + "cot": "20*2=40\n40*50=2000\nAnswer: 2000" + }, + { + "idx": 2131, + "label": "hard", + "cot": "1. Area of the square is $2 \\times 2 = 4$ sq in.\n2. Area of the right triangle is $\\frac{1}{2} \\times 2 \\times 2 = 2$ sq in.\n3. $4 / 2 = 2$, so at most 2 triangles can fit.\n\nAnswer: 2" + }, + { + "idx": 5993, + "label": "hard", + "cot": "1. Calculate daily calorie deficit: $2300 - 1800 = 500$ calories.\n2. Calculate total calories to burn: $10 \\text{ pounds} \\times 4000 \\text{ calories/pound} = 40000$ calories.\n3. Calculate days needed: $40000 / 500 = 80$ days.\n\nAnswer: 80" + }, + { + "idx": 1387, + "label": "hard", + "cot": "54+45+0=99\n99/3=33\nAnswer: 33" + }, + { + "idx": 5510, + "label": "hard", + "cot": "Karl=25\nRyan=25+20=45\nBen=45-10=35\nTotal=25+45+35=105\nAnswer: 105" + }, + { + "idx": 475, + "label": "easy", + "cot": "40+28+L = 3*30\n68+L = 90\nL = 90-68\nL = 22\nAnswer: 22" + }, + { + "idx": 4337, + "label": "hard", + "cot": "2*4*22=176\n176-54=122\nAnswer: 122" + }, + { + "idx": 960, + "label": "hard", + "cot": "Cindy=5\nJan=5+2=7\nMarcia=2*7=14\nGreg=14+2=16\nAnswer: 16" + }, + { + "idx": 4888, + "label": "hard", + "cot": "150/2=75\n75-50=25\nAnswer: 25" + }, + { + "idx": 3863, + "label": "hard", + "cot": "2.5*5=12.5\n0.5*3=1.5\n12.5+1.5=14\nAnswer: 840" + }, + { + "idx": 3357, + "label": "easy", + "cot": "100/2=50\n50/5=10\nAnswer: 10" + }, + { + "idx": 4166, + "label": "easy", + "cot": "10*10+10*5=150\nAnswer: 150" + }, + { + "idx": 2925, + "label": "hard", + "cot": "40*20=800\n800-10=790\nAnswer: 790" + }, + { + "idx": 6862, + "label": "hard", + "cot": "500*30=15000\n15000*0.70=10500\n10500/4=2625\n\nAnswer: 2625" + }, + { + "idx": 2055, + "label": "easy", + "cot": "1. $7.20 * 5 = 36.00$\n2. $36.00 * 8 = 288.00$\nAnswer: 288.00" + }, + { + "idx": 1256, + "label": "hard", + "cot": "12+9+8=29\n29*4=116\nAnswer: 116" + }, + { + "idx": 4592, + "label": "hard", + "cot": "1. First expedition: 3 weeks.\n2. Second expedition: $3+2=5$ weeks; Last expedition: $2 \\times 5 = 10$ weeks.\n3. Total weeks: $3 + 5 + 10 = 18$ weeks; Total days: $18 \\times 7 = 126$.\n\nAnswer: 126" + }, + { + "idx": 5659, + "label": "hard", + "cot": "100*1.15=115\n115+15=130\n100+115+130=345\nAnswer: 345" + }, + { + "idx": 101, + "label": "hard", + "cot": "6*2=12\n12*3=36\nAnswer: 36" + }, + { + "idx": 6074, + "label": "hard", + "cot": "500/2=250\n250-5=245\nAnswer: 245" + }, + { + "idx": 647, + "label": "hard", + "cot": "40-18=22\n22-16=6\n6-4=2\n\nAnswer: 2" + }, + { + "idx": 2751, + "label": "hard", + "cot": "Carlos has 20 cards.\nMatias has $20-6=14$ cards.\nJorge has 14 cards, total is $14+14+20=48$.\nAnswer: 48" + }, + { + "idx": 6054, + "label": "hard", + "cot": "19*400=7600\n10000-7600=2400\n2400/1=2400\nAnswer: 2400" + }, + { + "idx": 374, + "label": "hard", + "cot": "Day 1 buyers: 50\nDay 2 buyers: 25\nDay 3 buyers: 65\nTotal buyers: 140\n\nAnswer: 140" + }, + { + "idx": 6243, + "label": "hard", + "cot": "50*1.12=56\n56+5=61\nAnswer: 61" + }, + { + "idx": 2885, + "label": "hard", + "cot": "27+14+21=62\n5+17=22\n62-22=40\nAnswer: 40" + }, + { + "idx": 2358, + "label": "hard", + "cot": "24*0.5=12\n20*0.9=18\n12+18=30\nAnswer: 30" + }, + { + "idx": 5516, + "label": "easy", + "cot": "9-3=6\n6+7=13\nAnswer: 13" + }, + { + "idx": 2942, + "label": "easy", + "cot": "150*12*5=9000\nAnswer: 9000" + }, + { + "idx": 7306, + "label": "hard", + "cot": "2*2=4\n2*2.50=5\n20-9.00=11\n\nAnswer: 11" + }, + { + "idx": 3394, + "label": "easy", + "cot": "16.5+26+24.5=67\n95-67=28\nAnswer: 28" + }, + { + "idx": 661, + "label": "hard", + "cot": "30*5/6=25\n30-25=5\nAnswer: 5" + }, + { + "idx": 12, + "label": "easy", + "cot": "60/2=30\n30-5=25\n60+25=85\nAnswer: 85" + }, + { + "idx": 1575, + "label": "hard", + "cot": "200*10=2000\n2000*20=40000\n40000-2000=38000\nAnswer: 38000" + }, + { + "idx": 1311, + "label": "easy", + "cot": "1700*0.15=255\n1700+255=1955\nAnswer: 1955" + }, + { + "idx": 1961, + "label": "hard", + "cot": "50*10=500\n4000/500=8\n8/1=8\nAnswer: 8" + }, + { + "idx": 1827, + "label": "hard", + "cot": "1. Eggs per day: $(2 \\times 4) + (3 \\times 1) + 2 = 8 + 3 + 2 = 13$\n2. Breakfast days per year: $5 \\times 52 = 260$\n3. Total eggs: $13 \\times 260 = 3380$\n\nAnswer: 3380" + }, + { + "idx": 5221, + "label": "hard", + "cot": "60-10=50\n50/5=10\nAnswer: 10" + }, + { + "idx": 3101, + "label": "hard", + "cot": "180/30=6\n6*40=240\nAnswer: 240" + }, + { + "idx": 7171, + "label": "hard", + "cot": "2*4=8\n8*5=40\n40*4=160\nAnswer: 160" + }, + { + "idx": 5519, + "label": "hard", + "cot": "30*12=360\n20*24=480\n15*12=180\n360+480+180=1020\n1020*2.00=2040.00\nAnswer: 2040.00" + }, + { + "idx": 7164, + "label": "hard", + "cot": "21/7=3\n2\n3+2=5\nAnswer: 5" + }, + { + "idx": 258, + "label": "hard", + "cot": "Anne is 80cm tall.\nSister is $80/2=40$cm tall.\nBella is $3 \\times 80=240$cm tall, difference is $240-40=200$cm.\nAnswer: 200" + }, + { + "idx": 3295, + "label": "hard", + "cot": "1. Karen and Donald have 8 people, Tom and Eva have 6 people, total 14 people.\n2. The number of people in the pool is 16 divided by 2 legs per person, which is 8 people.\n3. $14 - 8 = 6$.\nAnswer: 6" + }, + { + "idx": 7140, + "label": "hard", + "cot": "1. Convert sister's height to inches: $4 \\times 12 + 3 = 51$ inches.\n2. Calculate sunflower height in inches: $51 + 21 = 72$ inches.\n3. Convert sunflower height to feet: $72 / 12 = 6$ feet.\n\nAnswer: 6" + }, + { + "idx": 6325, + "label": "hard", + "cot": "6*3=18\n6*3=18\n18+18=36\n50-36=14\nAnswer: 14" + }, + { + "idx": 5424, + "label": "easy", + "cot": "9+10+10=29\n9+2*10=29\nAnswer: 29" + }, + { + "idx": 1357, + "label": "easy", + "cot": "80+40=120\n120-50=70\n70+70=140\nAnswer: 140" + }, + { + "idx": 5746, + "label": "hard", + "cot": "12*1=12\n4*3=12\n12+12=24\nAnswer: 24" + }, + { + "idx": 1291, + "label": "hard", + "cot": "14 + 24 = 38\n42 - 38 = 4\nAnswer: 4" + }, + { + "idx": 3656, + "label": "hard", + "cot": "6*60=360\n360/20=18\n360/45=8\n18-8=10\nAnswer: 10" + }, + { + "idx": 4321, + "label": "hard", + "cot": "5/2=2.5 (This step is not strictly necessary but shows the relationship)\nLee=10\nTino=34\nAnswer: 34" + }, + { + "idx": 7249, + "label": "easy", + "cot": "1. $450 + 650 = 1100$\n2. $1100 - 1000 = 100$\n3. $100$\n\nAnswer: 100" + }, + { + "idx": 3991, + "label": "hard", + "cot": "4*5*2=40\n5*5*2=50\n40+50=90\nAnswer: 90" + }, + { + "idx": 7438, + "label": "hard", + "cot": "1. Calculate total current expenses: $20000 + $5000 + $8000 = $33000.\n2. Calculate required total income: $33000 + $42000 = $75000.\n3. Calculate additional income needed: $75000 - $65000 = $10000.\n\nAnswer: 10000" + }, + { + "idx": 4599, + "label": "hard", + "cot": "150*250=37500\n37500/30=1250\nAnswer: 1250" + }, + { + "idx": 4947, + "label": "easy", + "cot": "3*5=15\n15*20=300\nAnswer: 300" + }, + { + "idx": 6188, + "label": "hard", + "cot": "1. Calculate income from each job: $(20*10) + (30*20) + (5*40) = 200 + 600 + 200 = 1000$\n2. Calculate net income: $1000 - 500 = 500$\n3. Final amount: $500$\n\nAnswer: 500" + }, + { + "idx": 0, + "label": "hard", + "cot": "48\n24\n72\nAnswer: 72" + }, + { + "idx": 4051, + "label": "hard", + "cot": "1. Calculate the cost of the Taco Salad: $10\n2. Calculate the cost of the hamburgers: $5 * 5 = $25\n3. Calculate the cost of the fries and lemonade: (4 * $2.50) + (5 * $2) = $10 + $10 = $20. Total bill: $10 + $25 + $20 = $55. Split the bill: $55 / 5 = $11.\nAnswer: 11" + }, + { + "idx": 2670, + "label": "easy", + "cot": "2*500=1000\n1000+500=1500\nAnswer: 1500" + }, + { + "idx": 4493, + "label": "hard", + "cot": "1. posts per minute = 60 / 10 = 6\n2. earnings per minute = 6 * 0.25 = 1.5\n3. earnings per hour = 1.5 * 60 = 90\n\nAnswer: 90" + }, + { + "idx": 683, + "label": "hard", + "cot": "40/2=20\n20-20/4=15\nAnswer: 15" + }, + { + "idx": 5943, + "label": "hard", + "cot": "42/2=21\n42-21=21\n21/3=7\nAnswer: 7" + }, + { + "idx": 1749, + "label": "hard", + "cot": "30*25=750\n750/15=50\n50*300-3000=12000\n\nAnswer: 12000" + }, + { + "idx": 6182, + "label": "easy", + "cot": "2*6=12\n2*7=14\n12+14=26\nAnswer: 26" + }, + { + "idx": 4328, + "label": "hard", + "cot": "8*3=24\n14*2=28\n24+28=52\nAnswer: 52" + }, + { + "idx": 5014, + "label": "hard", + "cot": "5*17=85\n7*10=70\n3*6=18\n2*20=40\n85+70+18+40=213\nAnswer: 213" + }, + { + "idx": 801, + "label": "hard", + "cot": "500*0.10=50\n500-50=450\n450-55-50=345\nAnswer: 345" + }, + { + "idx": 1560, + "label": "hard", + "cot": "Budget is 60.\nNew price is $60 * 1.20 = 72$.\nCost of smaller frame is $72 * 3/4 = 54$.\nMoney remaining is $60 - 54 = 6$.\n\nAnswer: 6" + }, + { + "idx": 974, + "label": "hard", + "cot": "4+1=5\n20/5=4\nAnswer: 4" + }, + { + "idx": 4983, + "label": "hard", + "cot": "1. Calculate total water needed per hour: $800 + 400 = 1200$ ml.\n2. Convert Violet's capacity to ml: $4.8 \\times 1000 = 4800$ ml.\n3. Calculate hiking time: $4800 / 1200 = 4$.\n\nAnswer: 4" + }, + { + "idx": 1626, + "label": "easy", + "cot": "7*5=35\n10+35=45\nAnswer: 45" + }, + { + "idx": 2477, + "label": "hard", + "cot": "12*9=108\n160-108=52\nAnswer: 52" + }, + { + "idx": 1492, + "label": "hard", + "cot": "8-3=5\n8+5=13\nAnswer: 13" + }, + { + "idx": 3896, + "label": "easy", + "cot": "60/2=30\n30+15=45\n60-45=15\nAnswer: 15" + }, + { + "idx": 3249, + "label": "hard", + "cot": "1. Cherries cost $2.20 * 6 = $13.20 per pound.\n2. Cost of strawberries is $5 * 2.20 = $11.00.\n3. Cost of cherries is $5 * 13.20 = $66.00; total cost is $11.00 + $66.00 = $77.00.\n\nAnswer: 77.00" + }, + { + "idx": 2250, + "label": "hard", + "cot": "20*1=20\n20*6=120\n120/16=7.5\nAnswer: 8" + }, + { + "idx": 3710, + "label": "hard", + "cot": "1. Total time for 4 students is $4 \\times 30 = 120$ seconds.\n2. Total time for the last three is $3 \\times 35 = 105$ seconds.\n3. Time for the first student is $120 - 105 = 15$ seconds.\n\nAnswer: 15" + }, + { + "idx": 6490, + "label": "easy", + "cot": "65+15=80\n80-4=76\nAnswer: 76" + }, + { + "idx": 948, + "label": "hard", + "cot": "William goes 2 times per week.\nJason goes $4 \\times 2 = 8$ times per week.\nJason goes $8 \\times 4 = 32$ times in 4 weeks.\nAnswer: 32" + }, + { + "idx": 2101, + "label": "hard", + "cot": "1. $10 \\times 2 = 20$\n2. $20 \\div 4 = 5$\n3. $5$\nAnswer: 5" + }, + { + "idx": 1092, + "label": "easy", + "cot": "24/3=8\n8+2=10\nAnswer: 10" + }, + { + "idx": 5354, + "label": "hard", + "cot": "55+101+78=234\n234/6=39\nAnswer: 39" + }, + { + "idx": 6695, + "label": "hard", + "cot": "1. Gas station cost: $1.75 * 24 = $42.00\n2. Online cost: $5.00 * 2 = $10.00\n3. Savings: $42.00 - $10.00 = $32.00\n\nAnswer: 32.00" + }, + { + "idx": 5283, + "label": "hard", + "cot": "40*1.05=42\n42-8=34\n34/2=17\n\nAnswer: 17" + }, + { + "idx": 7142, + "label": "hard", + "cot": "20*3=60\n90-60=30\nAnswer: 30" + }, + { + "idx": 1265, + "label": "hard", + "cot": "30*4+24*2=120+48=168\n168/6=28\nAnswer: 28" + }, + { + "idx": 2280, + "label": "hard", + "cot": "20/4=5\n20-5=15\n15*(1/3)=5\n\nAnswer: 5" + }, + { + "idx": 6973, + "label": "easy", + "cot": "1. $4000 * 3/4 = 3000$\n2. $3000 / 10 = 300$\n3. $300$\nAnswer: 300" + }, + { + "idx": 152, + "label": "easy", + "cot": "80*0.45=36\n80-36=44\nAnswer: 44" + }, + { + "idx": 1685, + "label": "easy", + "cot": "6.00 + 1.00 = 7.00\n7.00 * 5 = 35.00\nAnswer: 35.00" + }, + { + "idx": 5578, + "label": "easy", + "cot": "3*20=60\n60/2=30\nAnswer: 30" + }, + { + "idx": 4574, + "label": "easy", + "cot": "10+6=16\n16-2=14\nAnswer: 14" + }, + { + "idx": 6933, + "label": "easy", + "cot": "100*2=200\n500-200=300\n300/3=100\nAnswer: 100" + }, + { + "idx": 5745, + "label": "hard", + "cot": "1. Calculate the time for one cycle: $10 \\text{ min (stand)} + 5 \\text{ min (sit)} = 15 \\text{ min}$.\n2. Calculate the number of full cycles in 120 minutes: $120 / 15 = 8$.\n3. Since the 8th cycle ends exactly at 120 minutes, Barry completes 8 turns.\n\nAnswer: 8" + }, + { + "idx": 5368, + "label": "hard", + "cot": "30+60+42=132\n132/6=22\nAnswer: 22" + }, + { + "idx": 7395, + "label": "hard", + "cot": "84\n84*10=840\n840/2=420\nAnswer: 420" + }, + { + "idx": 3051, + "label": "hard", + "cot": "50*2+100*3+150*2\n100+300+300\n700\nAnswer: 700" + }, + { + "idx": 7138, + "label": "hard", + "cot": "315*3=945\n1250/7=178.57\n1250-945=305\n\nAnswer: 305" + }, + { + "idx": 4406, + "label": "hard", + "cot": "25+11=36\n36*4=144\nAnswer: 144" + }, + { + "idx": 1702, + "label": "easy", + "cot": "60*2/3=40\n60-40=20\nAnswer: 20" + }, + { + "idx": 2223, + "label": "hard", + "cot": "100/2=50\n50/2=25\n25-11=14\nAnswer: 14" + }, + { + "idx": 2731, + "label": "easy", + "cot": "98-27=71\n71-15=56\nAnswer: 56" + }, + { + "idx": 6468, + "label": "easy", + "cot": "2000*0.02=40\n2000+40=2040\nAnswer: 2040" + }, + { + "idx": 3008, + "label": "easy", + "cot": "10/10=1\n10-1=9\nAnswer: 9" + }, + { + "idx": 5885, + "label": "hard", + "cot": "1. First hour: $60 \\div 12 = 5$ shirts.\n2. Second hour: $60 \\div 6 = 10$ shirts.\n3. Total shirts: $5 + 10 = 15$.\n\nAnswer: 15" + }, + { + "idx": 767, + "label": "easy", + "cot": "40/4=10\n10/2=5\nAnswer: 5" + }, + { + "idx": 2208, + "label": "hard", + "cot": "6*4=24\n2*2=4\n24+4=28\nAnswer: 28" + }, + { + "idx": 1224, + "label": "hard", + "cot": "80*1.15=92\n92+28=120\n120*1.10=132\n\nAnswer: 132" + }, + { + "idx": 2956, + "label": "hard", + "cot": "20*0.8=16\n20-16=4\nAnswer: 4" + }, + { + "idx": 3234, + "label": "hard", + "cot": "124600/2=62300\n62300/10=6230\nAnswer: 6230" + }, + { + "idx": 941, + "label": "easy", + "cot": "3*2*80=480\nAnswer: 480" + }, + { + "idx": 1963, + "label": "hard", + "cot": "24000*0.12=2880\n2880*0.40=1152\nAnswer: 1152" + }, + { + "idx": 2522, + "label": "hard", + "cot": "L=15, S+L=23\nS = 23-15 = 8\n1L+3S = 15 + 3*8 = 15 + 24 = 39\nAnswer: 39" + }, + { + "idx": 7020, + "label": "hard", + "cot": "1. $1 \\text{ hour} \\times 4 \\text{ miles} = 4 \\text{ miles per walk}$\n2. $4 \\text{ days} \\times 4 \\text{ miles/day} = 16 \\text{ miles walked}$\n3. $(31 - 4) \\times 4 = 27 \\times 4 = 108 \\text{ miles}$\nAnswer: 108" + }, + { + "idx": 5969, + "label": "hard", + "cot": "100-10=90\n90*(2/3)=60\n60*(1/2)=30\nAnswer: 30" + }, + { + "idx": 2479, + "label": "hard", + "cot": "15*2=30\n45-30=15\n15/1.5=10\n\nAnswer: 10" + }, + { + "idx": 2691, + "label": "easy", + "cot": "36-8=28\n28/2=14\nAnswer: 14" + }, + { + "idx": 2451, + "label": "hard", + "cot": "7*2=14\n98/14=7\nAnswer: 7" + }, + { + "idx": 4594, + "label": "easy", + "cot": "A=30\nB=60\nC=40\nD=35\nTotal A+B+C+D=165\nUnregistered=185-165=20\nAnswer: 20" + }, + { + "idx": 3941, + "label": "hard", + "cot": "1. Calculate the average speed: $(36+34+38)/3 = 108/3 = 36$ mph.\n2. Find the difference between the required speed and the average speed: $40 - 36 = 4$ mph.\n3. The difference is how much faster he needs to go.\n\nAnswer: 4" + }, + { + "idx": 3883, + "label": "easy", + "cot": "Paul drinks 3 bottles per day.\nTwice Paul's amount is 2*3=6.\nDonald drinks 3 more than 6, so 6+3=9.\nAnswer: 9" + }, + { + "idx": 2815, + "label": "easy", + "cot": "Nora's current age is 10.\nTerry's age in 10 years will be $4 \\times 10 = 40$.\nTerry's current age is $40 - 10 = 30$.\nAnswer: 30" + }, + { + "idx": 1018, + "label": "hard", + "cot": "1. Total members are $3 \\times 2 + 5 + 2 = 13$.\n2. Ron gets a turn once every 13 weeks, so $52/13 = 4$.\n3. Ron gets to pick a new book 4 times a year.\n\nAnswer: 4" + }, + { + "idx": 3924, + "label": "hard", + "cot": "10+6+5+9=30\n30/15=2\n2*3.50=7.00\nAnswer: 7.00" + }, + { + "idx": 3494, + "label": "hard", + "cot": "24+80=104\n104/8=13\nAnswer: 13" + }, + { + "idx": 6019, + "label": "easy", + "cot": "100*3=300\n300-100=200\nAnswer: 200" + }, + { + "idx": 5628, + "label": "hard", + "cot": "60*3=180\n180+60=240\nAnswer: 240" + }, + { + "idx": 7326, + "label": "easy", + "cot": "20 + (5 * 7) = 55\n55\nAnswer: 55" + }, + { + "idx": 1275, + "label": "hard", + "cot": "20*10=200\n30*10=300\n(30/2)*10=150\n200+300+150=650\nAnswer: 650" + }, + { + "idx": 1364, + "label": "hard", + "cot": "14*450=6300\n120*5=600\n6300+600+200=7100\nAnswer: 7100" + }, + { + "idx": 712, + "label": "hard", + "cot": "3*1=3\n4*3=12\n2*5=10\n3+12+10=25\n50-25=25\nAnswer: 25" + }, + { + "idx": 6617, + "label": "easy", + "cot": "1. Increase amount is $102 - 85 = 17$.\n2. Percent increase is $(17 / 85) * 100 \\approx 20\\%$.\n3. $17 / 85 \\approx 0.2$, so $0.2 * 100 = 20$.\n\nAnswer: 20" + }, + { + "idx": 693, + "label": "hard", + "cot": "20-6=14\n14-9=5\n5-4=1\n\nAnswer: 1" + }, + { + "idx": 6142, + "label": "hard", + "cot": "1. First floor earnings: $3 \\times 15 = 45$\n2. Second floor earnings: $3 \\times 20 = 60$\n3. Third floor earnings: $2 \\times 15 \\times 2 = 60$\nAnswer: 165" + }, + { + "idx": 4251, + "label": "hard", + "cot": "1. Week 1: 2 miles.\n2. Week 2: $2 \\times 2 + 3 = 7$ miles.\n3. Week 3: $\\frac{9}{7} \\times 7 = 9$ miles; Week 4: $9 - 5 = 4$ miles.\n\nAnswer: 4" + }, + { + "idx": 3674, + "label": "hard", + "cot": "1. Let M be male attendees and F be female attendees.\n2. M + F = 120 and M = F + 4, so (F+4) + F = 120.\n3. 2F = 116, so F = 58; M = 58 + 4 = 62.\nAnswer: 62" + }, + { + "idx": 5852, + "label": "easy", + "cot": "4*15=60\n66-60=6\nAnswer: 6" + }, + { + "idx": 1779, + "label": "hard", + "cot": "4*365=1460\n2*1460=2920\n1460+2920=4380\nAnswer: 4380" + }, + { + "idx": 3465, + "label": "hard", + "cot": "5*5=25\n4*2=8\n74-25-8=41\nAnswer: 41" + }, + { + "idx": 3017, + "label": "hard", + "cot": "Mandy's age is 3.\nBrother's age is $4 \\times 3 = 12$.\nSister's age is $12 - 5 = 7$.\nDifference is $7 - 3 = 4$.\n\nAnswer: 4" + }, + { + "idx": 6366, + "label": "easy", + "cot": "40-3=37\n37-5+12=44\n44+10=54\nAnswer: 54" + }, + { + "idx": 6733, + "label": "hard", + "cot": "1. Area of one side is $5*4=20$ sq ft.\n2. Total area to paint is $20*2=40$ sq ft.\n3. Paint needed is $40/4=10$ quarts, so cost is $10*2=20$.\nAnswer: 20" + }, + { + "idx": 7074, + "label": "hard", + "cot": "1. Convert time to hours: $4 + 30/60 = 4.5$ hours.\n2. Calculate speed: $360 / 4.5 = 80$.\n3. Speed is 80 mph.\n\nAnswer: 80" + }, + { + "idx": 5750, + "label": "hard", + "cot": "20*30=600\n40*30=1200\n600+1200=1800\nAnswer: 1800" + }, + { + "idx": 7397, + "label": "easy", + "cot": "1000 - 200 = 800\n800 / 10 = 80\nAnswer: 80" + }, + { + "idx": 4307, + "label": "hard", + "cot": "100*3/4=75\n75*12=900\n900*400=360000\nAnswer: 360000" + }, + { + "idx": 5545, + "label": "hard", + "cot": "1. Cost per pot is $15.00 + 4 \\times 4.00 + 4 \\times 3.50 = 15.00 + 16.00 + 14.00 = 45.00$.\n2. Since there are 4 corners, the total cost is $45.00 \\times 4 = 180.00$.\n3. Answer: 180.00" + }, + { + "idx": 4072, + "label": "hard", + "cot": "8*347=2776\n6*375=2250\n2776-2250=526\nAnswer: 526" + }, + { + "idx": 1697, + "label": "hard", + "cot": "21-6=15\n6/2=3\n15-3=12\nAnswer: 12" + }, + { + "idx": 6532, + "label": "hard", + "cot": "1. Younger son's current age is $40 - 10 = 30$.\n2. Younger son's age in 30 years will be $30 + 30 = 60$.\n3. Answer: 60" + }, + { + "idx": 1768, + "label": "hard", + "cot": "20 + 2 + 1 = 23\n23 - (3 * 2) = 17\nAnswer: 17" + }, + { + "idx": 5107, + "label": "easy", + "cot": "1. $I = PRT \\implies 1500 = P \\times 0.12 \\times 1$\n2. $1500 = 0.12P$\n3. $P = 1500 / 0.12 = 12500$\n\nAnswer: 12500" + }, + { + "idx": 857, + "label": "easy", + "cot": "6/3=2\n4*2=8\n2+8=10\nAnswer: 10" + }, + { + "idx": 1621, + "label": "hard", + "cot": "2*2=4\n14*4=56\nAnswer: 56" + }, + { + "idx": 3096, + "label": "hard", + "cot": "1. Time difference is 8 hours ($11 \\text{ A.M.} - 3 \\text{ A.M.} = 8 \\text{ hours}$).\n2. Number of 2-hour intervals is $8 / 2 = 4$.\n3. Total temperature increase is $4 \\times 1.5 = 6$ degrees; final temperature is $50 + 6 = 56$.\n\nAnswer: 56" + }, + { + "idx": 4476, + "label": "hard", + "cot": "58\n58*3=174\n58+174=232\nAnswer: 232" + }, + { + "idx": 1239, + "label": "hard", + "cot": "Area is $8*7=56$ sq ft.\nCost of new floor is $56*1.25=70$.\nTotal cost is $50+70=120$.\nAnswer: 120" + }, + { + "idx": 4619, + "label": "hard", + "cot": "8*9=72\n72-50=22\nAnswer: 22" + }, + { + "idx": 5229, + "label": "hard", + "cot": "4*25=100\n500/4=125\n125/25=5\n\nAnswer: 5" + }, + { + "idx": 3467, + "label": "easy", + "cot": "1. Calculate the number of bills George has: $(25 - 15) \\times 1 = 10$ bills.\n2. Calculate the number of bills spent and remaining: $10 \\times 0.20 = 2$ spent; $10 - 2 = 8$ remaining.\n3. Calculate the total amount received: $8 \\times 1.5 = 12$.\n\nAnswer: 12" + }, + { + "idx": 7189, + "label": "hard", + "cot": "11+13=24\n24+1=25\n13/25*100=52\n\nAnswer: 52" + }, + { + "idx": 4058, + "label": "easy", + "cot": "Brother's age in 6 years is 30.\nBrother's current age is $30 - 6 = 24$.\nDog's current age is $24 / 4 = 6$.\nDog's age in 6 years is $6 + 6 = 12$.\n\nAnswer: 12" + }, + { + "idx": 7272, + "label": "hard", + "cot": "120/5=24\n4+5+6=15\n24-15=9\nAnswer: 9" + }, + { + "idx": 4084, + "label": "hard", + "cot": "12-2=10\n10*3/5=6\n10-6=4\nAnswer: 4" + }, + { + "idx": 7165, + "label": "hard", + "cot": "1000/2=500\n500/2=250\n1000-500-250=250\n\nAnswer: 250" + }, + { + "idx": 1654, + "label": "hard", + "cot": "225+125+225+125=700\n700-3-10=687\nAnswer: 687" + }, + { + "idx": 4446, + "label": "hard", + "cot": "1. Adults are $12 - 2 = 10$.\n2. Deposit is $20 + (10 \\times 3) + (2 \\times 1) = 20 + 30 + 2 = 52$.\n3. Hans has to pay $52.\n\nAnswer: 52" + }, + { + "idx": 79, + "label": "hard", + "cot": "40-10=30\n40+30=70\nAnswer: 70" + }, + { + "idx": 5780, + "label": "easy", + "cot": "2000*0.30=600\n2000-600=1400\nAnswer: 1400" + }, + { + "idx": 2607, + "label": "hard", + "cot": "8/2=4\n4-1=3\n3*600-300=1500\n\nAnswer: 1500" + }, + { + "idx": 290, + "label": "easy", + "cot": "1. Adults = 3; Girls = 7; Total people = 10.\n2. Eggs for adults = $3 \\times 3 = 9$; Eggs for girls = $7 \\times 1 = 7$; Remaining eggs for boys = $36 - 9 - 7 = 20$.\n3. Let $x$ be eggs for a girl (1), boys get $1+x$. $B(1+x) = 20 \\implies B(2) = 20 \\implies B = 10$." + }, + { + "idx": 4301, + "label": "hard", + "cot": "4*3=12\n2*3=6\n1*3=3\n12+6+3=21\nAnswer: 21" + }, + { + "idx": 1277, + "label": "easy", + "cot": "64/4=16\n16-3=13\nAnswer: 13" + }, + { + "idx": 3104, + "label": "hard", + "cot": "200*0.20=40\n40-10=30\nAnswer: 30" + }, + { + "idx": 4775, + "label": "hard", + "cot": "Matt's current age is 5.\nKaylee's age in 7 years will be $3 \\times 5 = 15$.\nKaylee's current age is $15 - 7 = 8$.\nAnswer: 8" + }, + { + "idx": 2411, + "label": "hard", + "cot": "1. $800 * 0.20 = 160$\n2. $800 - 160 = 640$\n3. $640 / 40 = 16$\n\nAnswer: 16" + }, + { + "idx": 6611, + "label": "hard", + "cot": "8*3=24\n6*2=12\n24+12=36\nAnswer: 36" + }, + { + "idx": 3853, + "label": "hard", + "cot": "2800/350=8\n8*0.25=2\nAnswer: 2" + }, + { + "idx": 543, + "label": "easy", + "cot": "25*50=1250\n4*25=100\n1250-100=1150\nAnswer: 1150" + }, + { + "idx": 4231, + "label": "hard", + "cot": "25*1/2=12.5\n4000/200=20\n12.5*20=250\n\nAnswer: 250" + }, + { + "idx": 1843, + "label": "hard", + "cot": "20/2=10\n300/10=30\nAnswer: 30" + }, + { + "idx": 332, + "label": "easy", + "cot": "60000*0.18=10800\n60000-10800=49200\nAnswer: 49200" + }, + { + "idx": 2461, + "label": "hard", + "cot": "20000-1000*6=14000\nAnswer: 14000" + }, + { + "idx": 125, + "label": "easy", + "cot": "20+3*10=50\n20+30=50\nAnswer: 50" + }, + { + "idx": 6216, + "label": "hard", + "cot": "1. Calculate the original budget: $15 \\times 5 = 75$.\n2. Calculate the new budget: $75 \\times 1.20 = 90$.\n3. Calculate the number of softballs: $90 \\div 9 = 10$.\n\nAnswer: 10" + }, + { + "idx": 1316, + "label": "hard", + "cot": "2*6=12\n12*(2/3)=8\nAnswer: 8" + }, + { + "idx": 1219, + "label": "hard", + "cot": "2*8=16\n6*2=12\n3*3=9\n16+12+9=37\nAnswer: 37" + }, + { + "idx": 3774, + "label": "easy", + "cot": "18\n18/6=3\n3*2=6\nAnswer: 6" + }, + { + "idx": 3041, + "label": "easy", + "cot": "125+223=348\n493-348=145\nAnswer: 145" + }, + { + "idx": 3131, + "label": "easy", + "cot": "16\n8\n24\n\nAnswer: 24" + }, + { + "idx": 4340, + "label": "easy", + "cot": "20*10=200\n20/2=10\n200+100=300\nAnswer: 300" + }, + { + "idx": 742, + "label": "hard", + "cot": "8*1.5=12\n12*7=84\nAnswer: 84" + }, + { + "idx": 4243, + "label": "easy", + "cot": "20000*0.10=2000\n20000+2000=22000\nAnswer: 22000" + }, + { + "idx": 625, + "label": "easy", + "cot": "14\n28\n42\n\nAnswer: 42" + }, + { + "idx": 1688, + "label": "hard", + "cot": "8.5 + 9.5 = 18\n18 pints * 2 cups/pint = 36\nAnswer: 36" + }, + { + "idx": 4386, + "label": "hard", + "cot": "30*4=120\n12*5=60\n120+60=180\nAnswer: 180" + }, + { + "idx": 4903, + "label": "hard", + "cot": "4700-4300=400\n400/2=200\nAnswer: 200" + }, + { + "idx": 3422, + "label": "hard", + "cot": "2*2000=4000\n4000/100=40\n40*10=400\nAnswer: 400" + }, + { + "idx": 6926, + "label": "hard", + "cot": "7\n14\n19\n\nAnswer: 19" + }, + { + "idx": 4803, + "label": "hard", + "cot": "2*2.00=4.00\n4.00+2.00+0.50+1.50=8.00\n8.00/8=1.00\nAnswer: 1.00" + }, + { + "idx": 7252, + "label": "hard", + "cot": "10*2.00=20\n10*(2.00+1.00)=30\n20+30=50\nAnswer: 50" + }, + { + "idx": 6068, + "label": "hard", + "cot": "25-8=17\n17/2=8.5\n17-8.5-2.5=6\n\nAnswer: 6" + }, + { + "idx": 1490, + "label": "easy", + "cot": "10-3-1-3=3\nAnswer: 3" + }, + { + "idx": 4042, + "label": "hard", + "cot": "8000+4000=12000\n12000/20=600\n300+600=900\nAnswer: 900" + }, + { + "idx": 2146, + "label": "easy", + "cot": "14*3=42\n42-10=32\nAnswer: 32" + }, + { + "idx": 2480, + "label": "easy", + "cot": "30-12=18\n18*2=36\nAnswer: 36" + }, + { + "idx": 3342, + "label": "hard", + "cot": "Don eats 80 pizzas.\nDaria eats $2.5 \\times 80 = 200$ pizzas.\nTotal eaten is $80 + 200 = 280$ pizzas.\nAnswer: 280" + }, + { + "idx": 3143, + "label": "hard", + "cot": "5 chocolate bars\n7*5=35 M&Ms\n6*35=210 marshmallows\n5+35+210=250 total candies\n250/10=25 baskets\nAnswer: 25" + }, + { + "idx": 1341, + "label": "easy", + "cot": "15*2=30\n60-30=30\nAnswer: 30" + }, + { + "idx": 1043, + "label": "hard", + "cot": "2500+3500+2000=8000\n8000*0.10=800\n8000-800=7200\nAnswer: 7200" + }, + { + "idx": 2351, + "label": "hard", + "cot": "4/5*40=32\n32*15=480\nAnswer: 480" + }, + { + "idx": 454, + "label": "hard", + "cot": "12+5+5+2=24\n30-24=6\nAnswer: 6" + }, + { + "idx": 294, + "label": "easy", + "cot": "95-14=81\n81-25=56\n56-6=50\nAnswer: 50" + }, + { + "idx": 7290, + "label": "hard", + "cot": "300/2=150\n150/2=75\nAnswer: 75" + }, + { + "idx": 5875, + "label": "easy", + "cot": "8 + 21 = 29\n29" + }, + { + "idx": 5517, + "label": "hard", + "cot": "2.5 hours * 60 min/hr = 150 min\n1.5 hours * 60 min/hr = 90 min\n150 + 90 = 240 min\n240 - 97 = 143 min\n\nAnswer: 143" + }, + { + "idx": 5723, + "label": "hard", + "cot": "160*3=480\n480-210=270\nAnswer: 270" + }, + { + "idx": 2889, + "label": "hard", + "cot": "1. Calculate the percentage of guests who responded: $83\\% + 9\\% = 92\\%$.\n2. Calculate the percentage of guests who did not respond: $100\\% - 92\\% = 8\\%$.\n3. Calculate the number of guests who did not respond: $200 \\times 0.08 = 16$.\n\nAnswer: 16" + }, + { + "idx": 3368, + "label": "hard", + "cot": "6*50=300\n3*300=900\n300+900=1200\nAnswer: 1200" + }, + { + "idx": 5011, + "label": "hard", + "cot": "40*0.25=10\n40-10=30\n30*5=150\nAnswer: 150" + }, + { + "idx": 3819, + "label": "hard", + "cot": "50-3=47\n47+5=52\n52-15=37\nAnswer: 37" + }, + { + "idx": 3857, + "label": "hard", + "cot": "80*0.30=24\n24/2=12\nAnswer: 12" + }, + { + "idx": 5057, + "label": "easy", + "cot": "21/3=7\n21-7=14\nAnswer: 14" + }, + { + "idx": 862, + "label": "hard", + "cot": "575/5=115\n130-115=15\nAnswer: 15" + }, + { + "idx": 5179, + "label": "hard", + "cot": "200*0.8=160\n160*8=1280\n80+1280=1360\nAnswer: 1360" + }, + { + "idx": 2427, + "label": "hard", + "cot": "210+420+150=780\n780/3=260\nAnswer: 260" + }, + { + "idx": 6459, + "label": "hard", + "cot": "Juliet's age is 10.\nMaggie's age is $10-3=7$ and Ralph's age is $10+2=12$.\n$7+12=19$.\nAnswer: 19" + }, + { + "idx": 1041, + "label": "easy", + "cot": "650*0.8=520\n650-520=130\nAnswer: 130" + }, + { + "idx": 6109, + "label": "hard", + "cot": "3+5+2=10\n10-4=6\nAnswer: 6" + }, + { + "idx": 6434, + "label": "hard", + "cot": "1*4=80\n80/20=4\n4*1.50=6\n10-6=4\nAnswer: 4" + }, + { + "idx": 3641, + "label": "easy", + "cot": "21-5=16\n16-7=9\nAnswer: 9" + }, + { + "idx": 3043, + "label": "hard", + "cot": "14+3*14=56\n56/2=28\nAnswer: 28" + }, + { + "idx": 6198, + "label": "hard", + "cot": "Keith=3\nLarry=3*3=9\nDanny=9+5=14\n3+9+14=26\nAnswer: 26" + }, + { + "idx": 7388, + "label": "hard", + "cot": "91 hours\n8*7=56 hours\n91-56=35\nAnswer: 35" + }, + { + "idx": 3982, + "label": "hard", + "cot": "Niraj = 80\nBrittany = 3 * 80 = 240\nAngela = 3 * Brittany = 3 * 240 = 720\nTotal = Niraj + Brittany + Angela = 80 + 240 + 720 = 1040\n\nAnswer: 1040" + }, + { + "idx": 2092, + "label": "easy", + "cot": "Eduardo taught 3 classes.\nFrankie taught $3 \\times 2 = 6$ classes.\n$3 + 6 = 9$.\nAnswer: 9" + }, + { + "idx": 4684, + "label": "hard", + "cot": "Let $x$ be the amount of rain in the first hour.\nThe rain in the second hour is $2x + 7$.\nThe total rain is $x + (2x + 7) = 22$.\n$3x + 7 = 22$.\n$3x = 22 - 7$.\n$3x = 15$.\n$x = 15 / 3$.\n$x = 5$.\n\nAnswer: 5" + }, + { + "idx": 1881, + "label": "easy", + "cot": "5*7=35\n35*4=140\nAnswer: 140" + }, + { + "idx": 6359, + "label": "easy", + "cot": "4*20=80\n80+2*20=120\nAnswer: 120" + }, + { + "idx": 7280, + "label": "hard", + "cot": "4*3=12\n12*5=60\nAnswer: 60" + }, + { + "idx": 4290, + "label": "hard", + "cot": "180/3=60\n2/3*180=120\n140-120=20\nAnswer: 20" + }, + { + "idx": 3441, + "label": "hard", + "cot": "100*3=300\n300+5=305\n305-100=205\nAnswer: 205" + }, + { + "idx": 4636, + "label": "hard", + "cot": "32+13+15=60\n60-20=40\nAnswer: 40" + }, + { + "idx": 688, + "label": "easy", + "cot": "180*0.75=135\n180-135=45\nAnswer: 45" + }, + { + "idx": 812, + "label": "hard", + "cot": "22+13=35\n22/2=11; 13*2=26\n35+11+26=72\nAnswer: 72" + }, + { + "idx": 3401, + "label": "hard", + "cot": "Gerald has 8 notebooks.\nJack has $8+13=21$ notebooks initially.\nJack has $21-5-6=10$ notebooks left.\nAnswer: 10" + }, + { + "idx": 6015, + "label": "hard", + "cot": "James' current age is $27+3=30$.\nMatt's age in 5 years will be $2 \\times (30+5) = 2 \\times 35 = 70$.\nMatt's current age is $70-5=65$.\nAnswer: 65" + }, + { + "idx": 3416, + "label": "hard", + "cot": "40+50+65+36+82+72+26=371\n371/7=53\nAnswer: 53" + }, + { + "idx": 4068, + "label": "hard", + "cot": "1. $3 \\times 8 = 24$\n2. $3 \\times 6 = 18$\n3. $24 + 18 = 42$\n\nAnswer: 42" + }, + { + "idx": 2069, + "label": "hard", + "cot": "80*0.30=24\n80-24=56\n56+6=62\nAnswer: 62" + }, + { + "idx": 2887, + "label": "hard", + "cot": "75-15=60\n60-20=40\nAnswer: 40" + }, + { + "idx": 576, + "label": "hard", + "cot": "15+20=35\n15+35=50\nAnswer: 50" + }, + { + "idx": 240, + "label": "hard", + "cot": "7*18=126\n6*18=108\n126-108=18\nAnswer: 18" + }, + { + "idx": 2849, + "label": "hard", + "cot": "Breakfast cost: $0.95 + $0.85 = $1.80\nLunch cost: $4.65 + $1.15 = $5.80\nDifference: $5.80 - $1.80 = $4.00\n\nAnswer: 4.00" + }, + { + "idx": 6801, + "label": "hard", + "cot": "18/36*64\n1/2*64\n32\nAnswer: 32" + }, + { + "idx": 1888, + "label": "hard", + "cot": "15*12=180\n200-180=20\nAnswer: 20" + }, + { + "idx": 6709, + "label": "hard", + "cot": "5*20=100\n3*12=36\n4*30=120\n100+36+120+5=261\n400-261=139\n\nAnswer: 139" + }, + { + "idx": 7319, + "label": "easy", + "cot": "1. Distance = 4 * 50 = 200 miles\n2. Time = 200 / 100 = 2 hours\n3. Answer: 2" + }, + { + "idx": 26, + "label": "easy", + "cot": "2 * 0.20 = 0.4\n0.4 * 1000 = 400\nAnswer: 400" + }, + { + "idx": 1677, + "label": "hard", + "cot": "1. $2 \\times 3.20 + 2 \\times 1.90 + 2 \\times 2.40 = 6.40 + 3.80 + 4.80 = 15.00$\n2. $18.00 - 15.00 = 3.00$\n3. Answer: 3.00" + }, + { + "idx": 7380, + "label": "hard", + "cot": "240*0.60=144\n144*0.50=72\nAnswer: 72" + }, + { + "idx": 6127, + "label": "easy", + "cot": "3*2=6\n6*2=12\n12*2=24\n24*2=48\nAnswer: 48" + }, + { + "idx": 2402, + "label": "easy", + "cot": "39/3 is not an integer, so we assume the problem implies rounding or an error in the problem statement, but we proceed with the calculation based on the fraction.\n$39 \\div 3 = 13$\n$39 - 13 = 26$\nAnswer: 26" + }, + { + "idx": 4954, + "label": "hard", + "cot": "610-49-81=480\n480/5=96\nAnswer: 96" + }, + { + "idx": 1907, + "label": "easy", + "cot": "240/10=24\n24/2=12\nAnswer: 12" + }, + { + "idx": 7038, + "label": "easy", + "cot": "7-2=5\n6-3=3\n5+3=8\nAnswer: 8" + }, + { + "idx": 1162, + "label": "easy", + "cot": "200-80=120\n120/20=6\nAnswer: 6" + }, + { + "idx": 3719, + "label": "easy", + "cot": "10*2=20\n20+20=40\n75-30=45\nAnswer: 45" + }, + { + "idx": 920, + "label": "hard", + "cot": "10*2=20\n5*10=50\n100*5=500\n20+50+500=570\nAnswer: 570" + }, + { + "idx": 3905, + "label": "hard", + "cot": "12*5=60\n60*5=300\nAnswer: 300" + }, + { + "idx": 2821, + "label": "hard", + "cot": "15 pounds * 16 oz/lb = 240 ounces\n240 ounces / 12 ounces/steak = 20 steaks\nAnswer: 20" + }, + { + "idx": 2967, + "label": "hard", + "cot": "175*3=525\n525*0.20=105\n105-50=55\n\nAnswer: 55" + }, + { + "idx": 3877, + "label": "easy", + "cot": "4.5 - 1 = 3.5\n3.5 / 0.7 = 5\nAnswer: 5" + }, + { + "idx": 2397, + "label": "hard", + "cot": "1. Nathan's total hours: $3 \\text{ hours/day} \\times 14 \\text{ days} = 42$ hours.\n2. Tobias's total hours: $5 \\text{ hours/day} \\times 7 \\text{ days} = 35$ hours.\n3. Total hours: $42 + 35 = 77$ hours.\n\nAnswer: 77" + }, + { + "idx": 7195, + "label": "easy", + "cot": "40+2.25*x=76\n2.25*x=36\nx=16" + }, + { + "idx": 4531, + "label": "hard", + "cot": "12*2+9*3=24+27\n24+27=51\nAnswer: 51" + }, + { + "idx": 1507, + "label": "hard", + "cot": "Fritz = 40\nSean = 40/2 + 4 = 24\nRick = 3 * 24 = 72\nAnswer: 96" + }, + { + "idx": 4368, + "label": "hard", + "cot": "Perimeter is $2 \\times (20 + 60) = 160$ feet.\nTape left is $250 - 160 = 90$ feet.\nAnswer: 90" + }, + { + "idx": 1025, + "label": "easy", + "cot": "800*0.20=160\n800-160=640\nAnswer: 640" + }, + { + "idx": 2587, + "label": "hard", + "cot": "4\n8\n24\nAnswer: 36" + }, + { + "idx": 6183, + "label": "easy", + "cot": "4*3=12\n12/2=6\nAnswer: 6" + }, + { + "idx": 7053, + "label": "hard", + "cot": "20-3=17\n4*3=12\n17+30-12=35\nAnswer: 35" + }, + { + "idx": 5752, + "label": "hard", + "cot": "C=30\nW=3*30=90\nM=2*90=180\nTotal=30+90+180=300\nAnswer: 300" + }, + { + "idx": 3414, + "label": "easy", + "cot": "3*4*2=24\n24*3=72\nAnswer: 72" + }, + { + "idx": 1165, + "label": "hard", + "cot": "2*20*40*3=4800\n4800*8=38400\nAnswer: 38400" + }, + { + "idx": 2441, + "label": "easy", + "cot": "30/3=10\n10/2=5\nAnswer: 5" + }, + { + "idx": 2032, + "label": "hard", + "cot": "250-120=130\n130+35=165\n165-15=150\nAnswer: 150" + }, + { + "idx": 3643, + "label": "easy", + "cot": "8*3=24\n12*3=36\n24+36=60\nAnswer: 60" + }, + { + "idx": 3024, + "label": "easy", + "cot": "7+10=17\n2*17=34\nAnswer: 34" + }, + { + "idx": 4335, + "label": "hard", + "cot": "5*12=60\n60*20=1200\nAnswer: 1200" + }, + { + "idx": 69, + "label": "hard", + "cot": "7*5=35\n35+14=49\nAnswer: 49" + }, + { + "idx": 2626, + "label": "hard", + "cot": "50.00 * 0.90 = 45.00\n5.50 * 2 = 11.00\n45.00 + 11.00 + 10.00 = 66.00\nAnswer: 66.00" + }, + { + "idx": 745, + "label": "hard", + "cot": "500*2*1=1000\n500*4*0.5=1000\n1000+1000=2000\nAnswer: 2000" + }, + { + "idx": 1519, + "label": "hard", + "cot": "15*10=150\n15*6=90\n15*14=210\n150+90+210=450\nAnswer: 450" + }, + { + "idx": 889, + "label": "hard", + "cot": "1. $800 - 240 = 560$\n2. $560 / 80 = 7$\n3. Answer: 7" + }, + { + "idx": 2268, + "label": "hard", + "cot": "1. Convert 6 hours to minutes: $6 \\times 60 = 360$ minutes.\n2. Calculate total worms found: $360 \\div 15 = 24$ worms.\n3. Calculate number of toads: $24 \\div 3 = 8$ toads.\n\nAnswer: 8" + }, + { + "idx": 918, + "label": "hard", + "cot": "20.5*10=205\n6*4.5=27\n205+27=232\n\nAnswer: 232" + }, + { + "idx": 5637, + "label": "hard", + "cot": "10*45=450\n5*100=500\n500-450=50\nAnswer: 50" + }, + { + "idx": 1260, + "label": "easy", + "cot": "66+12*3=102\n12*3=36\n66+36=102\nAnswer: 102" + }, + { + "idx": 6518, + "label": "hard", + "cot": "50+8=58\n58+8=66\n66+8=74\n74+8=82\n82+8=90\nAnswer: 90" + }, + { + "idx": 5755, + "label": "easy", + "cot": "Mary is half of John's age: $M = J/2$.\nJohn is twice Mary's age: $J = 2M$.\nJohn is half of Tonya's age: $J = T/2$.\n$J = 60/2 = 30$.\n$M = 30/2 = 15$.\nAverage age is $(J+M+T)/3 = (30+15+60)/3 = 105/3 = 35$.\nAnswer: 35" + }, + { + "idx": 3654, + "label": "hard", + "cot": "20*1.5=30\n30+20=50\n50/5=10\nAnswer: 60" + }, + { + "idx": 6288, + "label": "hard", + "cot": "27+37=64\n83-64=19\nAnswer: 19" + }, + { + "idx": 1431, + "label": "hard", + "cot": "120/3=40\nSpeed is 40 mph\n200/40=5\n\nAnswer: 5" + }, + { + "idx": 2031, + "label": "hard", + "cot": "Campbell = 35\nKelsey = 35 + 8 = 43\nMartin = 43 - 3 = 40\n\nAnswer: 40" + }, + { + "idx": 4287, + "label": "easy", + "cot": "25-8-12=5\n8*1+12*0.5=14\n25-14=11\n\nAnswer: 11" + }, + { + "idx": 5173, + "label": "easy", + "cot": "6*4=24\n4*4=16\n24+16=40\nAnswer: 40" + }, + { + "idx": 3959, + "label": "easy", + "cot": "1. Find the price per kg: $6 / 2 = 3$.\n2. Calculate the cost for 12 kg: $12 * 3 = 36$.\nAnswer: 36" + }, + { + "idx": 1668, + "label": "hard", + "cot": "1. $60 / 4 = 15$ pounds of rocks.\n2. $15 \\text{ pounds} / 1.5 \\text{ pounds/rock} = 10$ rocks.\n3. Answer: 10" + }, + { + "idx": 2414, + "label": "hard", + "cot": "1. Calculate speed: $2000 \\text{ m} / 2 \\text{ h} = 1000 \\text{ m/h}$.\n2. Calculate time: $10000 \\text{ m} / 1000 \\text{ m/h} = 10 \\text{ hours}$.\n3. Convert to hours/minutes if needed (not required by prompt, but $10$ is the answer).\n\nAnswer: 10" + }, + { + "idx": 3678, + "label": "hard", + "cot": "1. $23 - 5 = 18$\n2. $18 / 3 = 6$\n3. $1 + 6 = 7$\nAnswer: 7" + }, + { + "idx": 2445, + "label": "hard", + "cot": "Eyes = 3\nWrinkles = 3 * 3 = 9\nSpots = 7 * 9 = 63\nCombined spots and wrinkles = 63 + 9 = 72\nDifference = 72 - 3 = 69\n\nAnswer: 69" + }, + { + "idx": 937, + "label": "easy", + "cot": "50*0.30=15\n50-15=35\nAnswer: 35" + }, + { + "idx": 4466, + "label": "hard", + "cot": "1. Total windows: $3 \\text{ windows/floor} \\times 3 \\text{ floors} = 9$ windows.\n2. Total earnings from windows: $9 \\text{ windows} \\times \\$2/\\text{window} = \\$18$.\n3. Deduction for time: $\\lfloor 6 \\text{ days} / 3 \\text{ days} \\rfloor \\times \\$1 = 2 \\times \\$1 = \\$2$.\n4. Total payment: $\\$18 - \\$2 = \\$16$.\n\nAnswer:" + }, + { + "idx": 4990, + "label": "easy", + "cot": "140 - (2 * 5) = 130\n10 - 2 = 8\nAnswer: 8" + }, + { + "idx": 5785, + "label": "hard", + "cot": "5\n4+5=9\n35-5-9-10=11 (Assuming autobiographies are 2 times fiction, $2 \\times 5 = 10$)\nAnswer: 11" + }, + { + "idx": 3861, + "label": "hard", + "cot": "2*400=800\n2*340=680\n800+680=1480\nAnswer: 1480" + }, + { + "idx": 758, + "label": "hard", + "cot": "5+28=33\n20+20+3=43\n43-33=10\nAnswer: 10" + }, + { + "idx": 5555, + "label": "hard", + "cot": "120*6=720\n120+720=840\nAnswer: 840" + }, + { + "idx": 989, + "label": "hard", + "cot": "7+8=15\n105/15=7\nAnswer: 7" + }, + { + "idx": 7239, + "label": "hard", + "cot": "10*4+5*5=65\n10*5+5*8=90\n65+90=155\nAnswer: 155" + }, + { + "idx": 4764, + "label": "easy", + "cot": "15*0.40=6\n15+6=21\nAnswer: 21" + }, + { + "idx": 3642, + "label": "hard", + "cot": "45-7=38\n38+45=83\nAnswer: 83" + }, + { + "idx": 698, + "label": "easy", + "cot": "16 * 0.5 = 8\n16 - 8 = 8\nAnswer: 8" + } +] \ No newline at end of file diff --git a/scripts/mcnemar_carrycot.py b/scripts/mcnemar_carrycot.py new file mode 100644 index 0000000..651965e --- /dev/null +++ b/scripts/mcnemar_carrycot.py @@ -0,0 +1,26 @@ +"""Paired McNemar: carry-cot arm vs feedforward control, both grid cells.""" +import json +from math import comb +from pathlib import Path + +OUT = Path(__file__).resolve().parent.parent / "results-loop" +A = json.load(open(OUT / "eval_gsm_carrycot_e400.json")) +B = json.load(open(OUT / "eval_gsm_carrycot_ff_e400.json")) +star = {it["idx"]: it["label"] + for it in json.load(open(OUT / "star_data.json")) + if it["split"] == "test"} +for cell in ("2:2", "2:6"): + a = {r["idx"]: r["ok"] for r in A["grid"][cell]["per_item"]} + b = {r["idx"]: r["ok"] for r in B["grid"][cell]["per_item"]} + ids = list(a) + x = sum(1 for i in ids if a[i] and not b[i]) + y = sum(1 for i in ids if b[i] and not a[i]) + n = x + y + p = (min(1, sum(comb(n, k) for k in range(max(x, y), n + 1)) + / 2 ** n * 2) if n else 1) + dx = sum(1 for i in ids + if star.get(i) == "drop" and a[i] and not b[i]) + dy = sum(1 for i in ids + if star.get(i) == "drop" and b[i] and not a[i]) + print(f"{cell}: carry-only={x} ff-only={y} McNemar p={p:.4f} " + f"drop-bucket discordants {dx}-{dy}") diff --git a/scripts/train_carry_cot.py b/scripts/train_carry_cot.py index ff36ae0..4765289 100644 --- a/scripts/train_carry_cot.py +++ b/scripts/train_carry_cot.py @@ -38,11 +38,42 @@ P_BY_LABEL = {"easy": 2, "hard": 6} ap = argparse.ArgumentParser() ap.add_argument("--feedforward", action="store_true") ap.add_argument("--seed", type=int, default=0) +ap.add_argument("--lensnoise", default=None, metavar="RANK,SCALE", + help="E2-N1: inject noise into the carried state during " + "training, shaped by the top-RANK sensitivity " + "directions of jbar at the band entrance; noise norm " + "= SCALE * per-position state norm (e.g. 32,0.05)") ARGS = ap.parse_args() TAG = ("carrycot_ff" if ARGS.feedforward else "carrycot") + ( + f"_ln{ARGS.lensnoise.replace(',', '_')}" if ARGS.lensnoise else "") + ( f"_s{ARGS.seed}" if ARGS.seed else "") +class LensNoiseWrapper(torch.nn.Module): + """Perturb the carried state s (not the anchor e) before the merge, + within the span of the lens's top-r readout-sensitive directions. + Train-time only (noise_on flag); eval and checkpoints use .base.""" + + def __init__(self, base, jbar_layer, rank, scale): + super().__init__() + self.base = base + self.scale = scale + self.noise_on = True + J = jbar_layer.float() + _, _, Vt = torch.linalg.svd(J, full_matrices=False) + self.register_buffer("V", Vt[:rank].T.contiguous()) # (d, r) + + def forward(self, e, s): + if self.noise_on and self.scale > 0: + z = torch.randn(*s.shape[:-1], self.V.shape[1], + device=s.device, dtype=torch.float32) + n = z @ self.V.T + n = n * (s.float().norm(dim=-1, keepdim=True) * self.scale + / (n.norm(dim=-1, keepdim=True) + 1e-6)) + s = (s.float() + n).to(s.dtype) + return self.base(e, s) + + def lr_at(step): if step < WARMUP: return LR * (step + 1) / WARMUP @@ -107,6 +138,15 @@ def main(): looper = BandLooper(model) adapter = MergeAdapter( d=model.config.get_text_config().hidden_size).cuda() + if ARGS.lensnoise: + r, sc = ARGS.lensnoise.split(",") + jbar = torch.load(Path(__file__).resolve().parent.parent + / "results/jbar.pt", map_location="cpu")["Jbar"] + from loop_common import BAND + adapter = LensNoiseWrapper(adapter, jbar[BAND[0]], int(r), + float(sc)).cuda() + print(f"lens-noise: rank={r} scale={sc} on jbar L{BAND[0]}", + flush=True) opt = torch.optim.AdamW(adapter.parameters(), lr=LR, weight_decay=0.01) keep = [it for it in data @@ -146,11 +186,15 @@ def main(): print(f"step {step:4d} {lbl:4s} loss={loss.item():.4f} " f"({(time.time()-t0)/(step+1):.1f}s/step)", flush=True) if step % 200 == 199 or step == STEPS - 1: + if ARGS.lensnoise: + adapter.noise_on = False for l in ("easy", "hard"): v = val_loss(looper, adapter, tok, val[l], P_BY_LABEL[l]) print(f" val@{step}: {l}={v:.3f}", flush=True) - torch.save(adapter.state_dict(), - OUT / f"adapter_{TAG}_e{step+1}.pt") + if ARGS.lensnoise: + adapter.noise_on = True + sd = (adapter.base if ARGS.lensnoise else adapter).state_dict() + torch.save(sd, OUT / f"adapter_{TAG}_e{step+1}.pt") json.dump(log, open(OUT / f"train_{TAG}_log.json", "w")) print("done", flush=True)