[jonfroehlich]

Runs claude-opus-5-5 (new) and claude-fable-5-1 (previously annapolis only) zero-shot on all 14 splits, juneau included. Both run at effort low on the first-party API. Everything was paid from the three subscriptions' monthly API credits, and each ledger row records the org that paid (api_account). The write-up is in docs/claude_newest_ids.md.

Results (paired 95% CIs are a pano bootstrap, RampNet − model)

• Opus 5.5 is now the strongest challenger.
  • It scores 0.712 pooled F1 over the nine US cities, and RampNet leads it by 0.081. The previous best challenger was YOLO11l, 0.193 behind.
  • Fable 5.1 pools 0.607.
• RampNet tops 11 of 14 splits.
  • Opus 5.5 beats it, with intervals that exclude zero, on laurens_mapillary (−0.136 [−0.216, −0.053]) and bayonne (−0.102 [−0.165, −0.040]).
  • Opus 5.5 on laurens_gsv and Fable 5.1 on bayonne score higher, but their intervals include zero, so they are reported as point estimates.
  • RampNet's lead over Opus 5.5 excludes zero on all 11 splits it tops; the narrowest is morgantown, +0.050 [+0.005, +0.097].
• Juneau (Juneau, AK benchmark split (Mapillary, Trimble MX7): staging, GT review, scoring #254, pooled by Juneau (#254): reviewed, scored, registered as the ninth pooled US split #260): RampNet 0.803; Opus 5.5 0.727 (+0.076 [+0.028, +0.126]); Fable 5.1 0.673 (+0.131 [+0.080, +0.182]).
• Opus 5.5 beats Fable 5.1 on all 14 splits.
  • The narrowest margin is juneau, +0.054.
  • Opus 5.5 uses about the same tokens per call and is about 2.5x cheaper per pano.
  • Loosening the match radius to 2x closes the gap by −6% to 53% (median about 40%). The rest is recall.
• Caveat: every Claude number is a single run, with no seed variance, at effort low only.

Cost (list-price ledger estimates; the billing console is authoritative)

• claude5: $95.35
• claude-team: $450.99
• claude20: $29.06 (juneau)
• Campaign total: $575.40. The Team org's purchased credit was not touched.

Changes

• pricing.py: claude-opus-5-5, read 2026-10-19, with a pinned test.
• compare.py:
  • --api-account, with a fixed list of choices.
  • Exits 3 when a leg is not runnable or aborts. The ledger row is still written first.
• A roster entry for Opus 5.5.
• run_claude_legs_by_account.ps1, the per-org runner with a ledger budget cap.
  • It stops on a non-zero exit.
  • It is safe under PowerShell 5.1 and 7.
  • It defaults to the main checkout.
• scripts/analysis/claude_newest_ids_261.py: paired CIs and the radius check. It reads only committed files, has a --check mode and is registered in check_all.py.
• 28 published detection files, all --verify'd identical to the cache, plus the ledger rows.
• Main merged in (brings Juneau (#254): reviewed, scored, registered as the ninth pooled US split #260). The scoreboard and derived artifacts are regenerated.
• Dated "superseded" notes on every claim the new legs made false. The UChicago talk docs are kept as delivered on 2026-10-16.
• Records Jon's 2026-10-19 decision to run Claude on manual_gold, reversing claude-opus-5 on manual_gold: recommend NOT running it alone, because the top challenger is not on that split either #144.

Review

1. Independent review: Newest Claude ids on every split: Opus 5.5 and Fable 5.1 #261 (review)
2. Fixes: d05bf4e, 08f7526, e4d48f9, 08d4ac6.
3. Final re-review: ready to merge. Its text-only follow-ups are in ece4fc4.
4. Test results: pytest -q 3,353 passed, 6 skipped. check_all.py --ci 22 passed.

🤖 Generated with Claude Code — Opus 5.5, claude-opus-5-5

[jonfroehlich]

Independent review: code + data/numbers/docs

Two independent reviewers: one on the code, one on the data, numbers and docs. Each finding was checked against the repo. No blockers. The data reproduces. The problems are in the launcher script and in prose that claims more than the data supports.

Independently confirmed

• Ledger. The 26 new rows are append-only and are the only rows carrying api_account. Each row's est_cost_usd equals pricing.estimate_cost exactly.
  • Per org: claude5 $95.35 (including the $0.33 calibration); claude-team $450.99.
  • Bayonne legs: $8.53 / $21.24.
  • Thinking tokens per call: Opus 5.5 18.1, Fable 5.1 2.6.
• Detections. All 26 files cover every pano in their split's records.jsonl (0 missing, 0 extra, 0 uncached), with consistent metadata. Bayonne panos match imagery_manifest.json (125/125).
• Scores. An independent greedy-match scorer reproduces every P/R/F1 and every tp/fp/fn in the write-up table.
  • Pooled: Opus 5.5 0.7098, Fable 5.1 0.5985, RampNet lead 0.082.
  • RampNet tops 10 of 13 splits.
• Regeneration. scoreboard.py regenerates scoreboard.json and all 5 figures byte-identically. The --check modes of laurens_paired_151, eval_protocol_150 and cascade_cost_35 pass.
• Tests. Full suite on Windows: 3,297 passed, 6 skipped. No assertion was meaningfully loosened.
• Cache keys. Unaffected: api_account goes only into report_usage, not into the detector signature, so no paid detections are orphaned.

Should-fix

1. The launcher's "stop on failure" never fires (run_claude_legs_by_account.ps1:70, compare.py:871/:952).
   • compare.py exits 0 when a leg is "not runnable" and when it aborts after 10 consecutive failures.
   • So if credits run out mid-run, every remaining split aborts in turn and leaves a partial leg. Nothing is overspent, but the STOP contract is false.
2. Under Windows PowerShell 5.1, $ErrorActionPreference='Stop' + 2>&1 on the native call kills compare.py on its first stderr line (reproduced). Its finally: report_usage never runs, so calls already paid for get no ledger row (the #139 failure mode). Fix: #Requires -Version 7.2, or relax the preference around the call.
3. Two "beats RampNet" claims have paired CIs that include zero. These come from the repo's bayonne_paired_159 pano bootstrap (10k draws, seed 0), figures as RampNet − challenger.
   • Opus 5.5 / laurens_gsv: −0.042 [−0.101, +0.020].
   • Fable 5.1 / bayonne: −0.023 [−0.096, +0.050].
   • Two do survive: Opus 5.5 / bayonne −0.102 [−0.165, −0.040] and Opus 5.5 / laurens_mapillary −0.136 [−0.216, −0.053].
   • The docs (write-up, scoreboard, comparison, findings row 1, report, bayonne doc, roster notes) state all four flat. Add the CIs and reword the two non-significant cases as point estimates.
4. The radius analysis overclaims (claude_newest_ids.md, "Why Opus 5.5 does better").
   • "About half" is 29–51% on the 3 splits used; the median over all 13 is 39%. On clovis the gap grows.
   • Three table cells are differences of rounded values.
   • The re-run command needs .model_cache. The committed detection files suffice, so use them and commit the script.
5. Juneau. benchmark/juneau/ is on main, and open #260 adds it to the US pool. Whichever PR lands second leaves Opus 5.5 / Fable 5.1 at 8/9 pooled coverage, which changes 0.710 / 0.082 / "all 13 splits". The dependency is not mentioned.
6. Stale model_comparison.md passages.
   • The manual_gold "0.34 widest lead" is now 0.186 against Opus 5.5 (0.722).
   • The supervised-baseline "splits 5–5 against the zero-shot field" is now 2–8, which contradicts this PR's own scoreboard finding 1.
7. laurens_paired_151.md §3.1 says "Three lose with an interval that excludes zero". Fable 5.1 (−0.135 [−0.225, −0.043]) makes four, and it is the largest loss.
8. The talk figure script's caption can name a model the chart doesn't show (uchicago_2026_figures.py). best_zs is now Opus 5.5 (0.71), but COMPARISON_MODELS still plots Opus 5 (0.57).

Nits

• Write-up overclaims its tests. It says every P/R/F1 and cost is "re-derived by tests/test_claude_published_legs.py". The tests cover only the r=0.022 table and the four org/leg totals.
• No pinned test for Opus 5.5 rates (Fable has one). The rates themselves are right.
• Ledger dollars are unreconciled estimates. Say so, since the billing console is authoritative.
• "Emits ~17% more points (317 vs 270)". The files emit 328 vs 285 (+15%); 317/270 are scored points.
• Talk README says "need Jon's decision before the talk". The talk was given 2026-10-16, before these runs.
• Heading "…not effort". Both models ran at effort low but with ~7× different thinking, so "not the effort setting" is the accurate wording.
• model_comparison.md:655 re-affirms 0.113. Unrounded it is 0.1139 (0.114 elsewhere).
• rampnet1_findings.md:44 paid-API row. It says $131.45; the ledger now totals $683.58.
• No replication pointer for the paid legs. laurens_gsv, laurens_mapillary and bayonne panos are not on HF (replication.md:31).
• Launcher script:
  • It can write the ledger into a worktree, and its Select-String filter hides compare.py's WARNING line.
  • The cap sums all-time spend rather than monthly; this fails safe.
  • -Splits a,b under -File binds as one string.
  • python is resolved from PATH.
• --api-account accepts any text. A typo silently creates a new bucket.
• scoreboard_render.py:789-792 mixes a display name with a LOG_LABEL id in one sentence (on a branch not reached today).

Next: fixes for all of the above, then a final independent re-review before merge.

🤖 Generated with Claude Code — Opus 5.5, claude-opus-5-5

[jonfroehlich]

Review fixes pushed (d05bf4e, 08f7526, e4d48f9, 08d4ac6). Every finding in the review above is addressed. Full suite: 3,336 passed, 6 skipped. The --check modes of scoreboard, laurens_paired_151, bayonne_paired_159 and eval_protocol_150 all pass, and so does the new claude_newest_ids_261 check.

Code (d05bf4e)

• Launcher stop never fired:
  • compare.py now exits 3 (EXIT_INCOMPLETE) when any leg was not runnable or aborted after consecutive failures, and prints an INCOMPLETE: block. Ledger rows are still written first, in each leg's finally.
  • Isolated per-pano failures still exit 0 by design. A re-run fills them from cache, and one 529 shouldn't strand every later split.
  • Every caller was checked. cascade_transfer_35.slurm (set -eo pipefail loop) got || echo ... so it doesn't end early. Tests cover not-runnable, aborted (with a ledger row) and isolated-failure exits.
• PS 5.1 stderr kill: $ErrorActionPreference='Continue' is now scoped around the native call (try/finally). Verified under both powershell.exe and pwsh with a fake Python child: the old script reproduced the kill, and the new one runs and writes the row.
• Talk caption: comparison_caption_facts() names the best zero-shot model among the bars, and lists stronger unshown models. Committed PNGs are untouched (as delivered 2026-10-16). On the scoreboard as of the talk it reproduces the delivered captions exactly.
• Nits:
  • Launcher: -DataRoot defaults to the main checkout; -Python defaults to the main venv; -Splits 'a,b' works under -File; WARNING lines pass the filter; the cap comment now says all-time.
  • --api-account has choices=.
  • Opus 5.5 rates now have a pinned test.
  • scoreboard_render uses the display name.

Data / docs (08f7526, e4d48f9, 08d4ac6)

• New scripts/analysis/claude_newest_ids_261.py (--check / --markdown, registered in check_all.py). It reads only committed bundles and benchmark/model_detections/, with no .model_cache.

• Paired CIs (RampNet − challenger, 10k pano bootstrap, seed 0). Reproduced the review's numbers exactly:

  Claim	Paired difference [95% CI]	Reading
  Opus 5.5, bayonne	−0.102 [−0.165, −0.040]	win
  Opus 5.5, laurens_mapillary	−0.136 [−0.216, −0.053]	win
  Opus 5.5, laurens_gsv	−0.042 [−0.101, +0.020]	point estimate only
  Fable 5.1, bayonne	−0.023 [−0.096, +0.050]	point estimate only

  • RampNet's lead over Opus 5.5 excludes zero on all 10 splits it leads; the narrowest is morgantown, +0.050 [+0.005, +0.097].
  • The wording is updated everywhere the claims appeared, with "single run, no seed variance, effort low only" alongside.

• Radius: at 2× radius the gap closes −6% to 53% over all 13 splits, median 39%. On clovis it grows. The gaps are now differences of unrounded F1s, which moves five main-table cells by 0.001.

• model_comparison.md:

  • manual_gold lead over the zero-shot field is now 0.187 [0.174, 0.200]. It is no longer the widest; budapest is 0.193.
  • Supervised baseline vs zero-shot is 2–8 with Opus 5.5 included.
  • 0.113 → 0.114.

• laurens_paired_151.md §3.1: four lose; Fable 5.1 at −0.135 [−0.225, −0.043] is the largest.

• Nits:

  • New tests/test_claude_newest_ids_261.py (26 tests) pins the radius table, point counts (328 vs 285 emitted), cost ranges, thinking stats and bayonne costs. The status line now says exactly what is tested.
  • Ledger dollars are labelled unreconciled list-price estimates.
  • The findings/report paid-API total is now $683.58 as of 2026-10-20.
  • Added a replication note that the laurens and bayonne panos are not on HF.

• Talk README: marked delivered 2026-10-16, figures kept as given, and the claims described as superseded 2026-10-19/20.

Still to come before the final re-review

#260 has merged. Opus 5.5 and Fable 5.1 are now running on juneau (125 panos, claude20 org, ~$30 estimated), so both models will have full 9-of-9 pooled coverage. Once they finish: merge main, publish the two legs, regenerate, then the final independent re-review.

🤖 Generated with Claude Code — Opus 5.5, claude-opus-5-5

[jonfroehlich]

Final re-review (head 0acfa93): Ready to merge

Two independent re-reviewers checked this head, one on code and tests and one on data, numbers and docs. Neither found a blocker, and the code reviewer found no should-fix items either. The data reviewer found five stale counts that leave out juneau; those are fixed in ece4fc4 along with the comment nits. That commit is text only and changes no number.

Since the first review

• a4e76d4 merges main, which brings in Juneau (#254): reviewed, scored, registered as the ninth pooled US split #260 (juneau as the ninth pooled US split).
• 0acfa93 adds the juneau legs for Opus 5.5 and Fable 5.1. They ran on the claude20 org on 2026-10-20 and cost $8.40 (2,610 s) and $20.66 (2,940 s). Each made 750 calls with no failures.

Results on juneau

Model	P	R	F1	RampNet − model [95% CI]
RampNet	0.967	0.687	0.803	—
Opus 5.5	0.733	0.721	0.727	+0.076 [+0.028, +0.126]
Fable 5.1	0.713	0.636	0.673	+0.131 [+0.080, +0.182]

Headline numbers, all independently recomputed

• Pooled F1 over 9 US cities: RampNet 0.793, Opus 5.5 0.712 (lead 0.081), Fable 5.1 0.607, YOLO11l 0.583, Gemini 3.1 Pro 0.571.
• RampNet tops 11 of 14 splits.
  • Opus 5.5 wins two with paired CIs that exclude zero: laurens_mapillary −0.136 [−0.216, −0.053] and bayonne −0.102 [−0.165, −0.040].
  • Two leads are point estimates only: Opus 5.5 on laurens_gsv and Fable 5.1 on bayonne.
• RampNet's lead over Opus 5.5 excludes zero on all 11 splits it tops.
• Spend:
  • Campaign: $575.40 (claude5 $95.35, claude-team $450.99, claude20 $29.06). These are list-price ledger estimates; the billing console is authoritative.
  • Whole ledger: $718.24.
• Ledger merge: append-only. No row from either side was dropped or duplicated.

Code re-review

• The original findings are fixed, verified by running them.
  • compare.py exit 3: the ledger row is written before the exit, every caller was audited, and the slurm set -e loop is protected.
  • The PS1 launcher was run under both PowerShell 5.1 and 7 with a fake child process. The old script reproduced the stderr kill; the new one stops on exit 3 with both rows written.
  • --api-account now takes a fixed list of choices, and the talk caption logic is fixed.
• No test was weakened relative to main.
• claude_newest_ids_261.py pairs its bootstrap draws correctly, reads only committed files, and its --check detects drift.

Checks

• Full suite: 3,353 passed, 6 skipped.
• check_all.py --ci: 22 passed, 0 failed (re-run on ece4fc4).
• scoreboard.py --check: current.

🤖 Generated with Claude Code — Opus 5.5, claude-opus-5-5