[jonfroehlich]

Stacked on #261. Base is claude-newest-ids; merge this after #261. (When #261 merges, GitHub retargets this to main.)

Jon asked (2026-10-20) for an aggregate column on the "F1 by model and split" heatmap (docs/figures/scoreboard_by_split.png) that the whole table is sorted by.

What changed

• Pooled mean column, at the right end of the pooled-city block and before the gutter. It is framed, set in bold, and has a bold "Pooled mean ▼" header. The value is the board's own pooled F1, the headline number (RampNet 0.793, Opus 5.5 0.712). It is read from the summary rather than recomputed, and by_split_rows() asserts that it equals the mean of the row's per-split cells (tolerance 2e-6, which covers the six-place rounding in scoreboard.json). It prints three decimals to match the headline. Cells keep two.
• Rows sorted by it. The 14 legs with 9/9 pooled coverage sort descending, and RampNet comes first because it sorts there, not because it is pinned. The 8 partial-coverage legs form a separate group below a labelled rule and sort by the mean over the cities they ran, shown with coverage, e.g. 0.533 (5/9). So a one-city leg is never ranked against a nine-city mean.
• Column-group labels ("Pooled US cities (nine)", "Pooled mean ▼", "Held out †") with hairlines above the grid, so the gutter reads as a boundary rather than a missing column.
• Colour ramp unchanged and values printed in every cell. The aggregate column is marked by its frame, bold type and position, not by a new hue.
• A footer line defines the pooled mean and the sort.

Row order, full group: RampNet 0.793, Claude Opus 5.5 0.712, Claude Fable 5.1 0.607, YOLO11l 0.583, Gemini 3.1 Pro 0.571, YOLO11x 0.551, YOLO26 0.547, Gemini 3.7 Flash 0.538, Gemini 3.6 Flash 0.525, Molmo2-8B 0.422, Qwen3-VL-32B 0.322, Qwen3-VL-8B 0.314, OWLv2 0.065, Grounding DINO 0.053.
Partial group: Claude Fable 5 0.611 (1/9), Claude Opus 5 low 0.568 (8/9), Mask2Former 1024 0.533 (5/9), Claude Opus 5 high 0.520 (1/9), Mask2Former curb cut 0.517 (1/9), Claude Sonnet 5 low 0.463 (1/9), Claude Sonnet 5 high 0.456 (1/9), Mask2Former +curb 0.210 (1/9).

Only the heatmap changes. The other four scoreboard figures regenerate byte-identical, and the markdown by-split table keeps its class-grouped order. docs/model_scoreboard.md gets a paragraph under the figure that explains the different row order.

Tests

• test_by_split_heatmap_rows_sort_by_pooled_mean_in_two_groups checks the grouping, the sort within each group, that the aggregate equals the board's pooled F1, and that RampNet is first (0.793). It reads the committed analysis_out/scoreboard.json and is CPU-only.
• test_by_split_rows_rejects_an_aggregate_that_disagrees_with_its_cells checks that a pooled F1 that drifts from its cells fails the assert.
• python -m pytest -q: 3354 passed, 7 skipped. scripts/check_all.py --ci: 22 passed, 0 failed, 3 skipped. scoreboard.py --check: current.

To regenerate: python scripts/analysis/scoreboard.py

🤖 Generated with Claude Code — Opus 5.5, claude-opus-5-5

[jonfroehlich]

Independent review (head 589a50c): Not ready, one factual fix

The code is sound. Verified:

• scoreboard.py --check exits 0.
• tests/test_scoreboard.py: 104 passed.
• A fresh render of the figure is pixel-identical to the committed PNG.
• The pooled-mean tolerance has margin: the largest drift is 5.6e-7 against a 2e-6 tolerance.
• Edge cases render correctly: a 0/9 leg, a partial-only board, a full-only board, and Mask2Former forced to 9/9 (simulating #267).
• Nothing breaks when #267 or #259 regenerates the figure.

Blocker

1. docs/model_scoreboard.md, Gemini note. The note says "the sweep used klone's ckpt partition, so the likeliest home was purged scratch". The repo's own log contradicts this. model_comparison.md (manual_gold "Cost and runtime", from b148f71) says "Gemini needs no GPU at all — both models ran from a desktop against Vertex"; only the GPU legs used ckpt. The cache was therefore desktop-local, and the two desktops (the Windows workstation and the Mac) are both among the machines searched. Fix: cite that line and drop the scratch guess.

Should-fix

2. The prose example 0.533 (5/9) will go stale. It is Mask2Former 1024, which #267 moves to full coverage. Use a leg that stays partial (0.568 (8/9)) or a generic form.
3. The new ordering test needs three changes:
   • mean_f1 == m["f1"] is tautological; drop it.
   • The 0.793 pin is a data value that breaks whenever the pool changes; drop it or derive it from the board.
   • The None / 0-coverage path is untested. Add a synthetic-board test covering a partial leg with a higher mean but lower coverage, a 0/9 leg, and a tie in the full group, and check that fig_by_split renders it.

Nits

4. Coverage assert format. The assert hard-codes the "n/N" coverage string format. Compare counts instead.
5. Fixed width. The figure width is fixed at 13 in. Scale it with the column count, as the height already scales with rows.
6. Partial-only caption. On a partial-only board the caption says "not ranked against the rows above" when there are no rows above. Guard the clause.
7. Held-out footer. It is still hard-coded and says "budapest" where the data says budapest_district5. Generate it from result["held_out"].
8. [-0.0, width] → 0.0.
9. A11y is fine. Every cell carries its number, and the frame and bold text mark the aggregate without using colour. One optional idea: give partial-row means a lighter fill.

Out of scope here, passed to #267: the fig_headline footer says "single-split legs", but the partial legs include 8/9 and 5/9 ones. And the scatter figures have no label nudge for Mask2Former 1024 once it goes full.

Fixes follow, then a re-review.

🤖 Generated with Claude Code — Opus 5.5, claude-opus-5-5

[jonfroehlich]

Review fixes pushed in fe76f90. Every finding in the review above is addressed. Checks: scoreboard.py --check passes; tests/test_scoreboard.py 106 passed; check_all.py --ci 22 passed. The only regenerated output that changed is the heatmap's footer text.

1. Gemini note. The note now says the cache was desktop-local and quotes model_comparison.md: "Gemini needs no GPU at all — both models ran from a desktop against Vertex". Only the GPU legs used ckpt. Both desktops were among the machines searched.
2. Prose example. Now 0.568 (8/9), which stays partial after Vistas 1024 parity arm on all fourteen splits (stacked on #261) #267.
3. Tests. The tautological check is dropped. The 0.793 pin is replaced by "RampNet is the best full-coverage mean". A new synthetic-board test covers coverage-first ordering, a 0/3 leg with None sorting last, and a tie keeping board order. It also renders fig_by_split for that board and for a partial-only board. A footer test is added too.
4. Coverage assert. It now compares counts, not the n/N string.
5. Figure width. It scales with the column count. At 14 splits it is unchanged: 12.97 in against 13.
6. Partial-only caption. The "not ranked against the rows above" clause appears only when full-coverage rows exist.
7. Held-out footer. It is generated from result["held_out"] and names budapest_district5.
8. -0.0. Now 0.0.

Not done: the lighter fill for partial means. The fill encodes magnitude, so lightening it would misstate the number. Partial rows are already marked without colour, by their separate group and the coverage label.

Next: a final independent re-review.

🤖 Generated with Claude Code — Opus 5.5, claude-opus-5-5

[jonfroehlich]

Final re-review (head fe76f90): Ready to merge

Every finding is verified as fixed:

• Gemini note. The quote matches model_comparison.md:2046 word for word and matches b148f71.
• Tests. Eight sort-key mutations each fail at least one test.
• Figure. A fresh render is pixel-identical to the committed figure.
• Held-out footer. Every generated reason is accurate for the five held-out splits.
• Checks. scoreboard.py --check passes; tests/test_scoreboard.py 106 passed; check_all.py --ci 22 passed.

Its three optional nits are applied in 10c93b4:

• The synthetic board now lists rows out of order and has two one-city partial legs, so the ordering test no longer depends on committed data.
• The literal budapest_district5 pin is dropped.
• The Gemini note now says why a desktop search can come up empty: compare.py's cache lives inside the checkout it ran from.

#267 also regenerates these figures. Whichever of the two merges second will merge main in and regenerate.

🤖 Generated with Claude Code — Opus 5.5, claude-opus-5-5