[jonfroehlich]

Completes the Mask2Former Vistas parity arm (vistas:curb-cut at --vistas-input-size 1024 1024, published as mask2former-vistas-curb-cut-1024x1024) on the nine splits it had not run: clovis, morgantown, laurens_mapillary, juneau, laurens_gsv, budapest_district5, sao_paulo, bayonne and manual_gold. It now covers all fourteen splits and scores in the pooled headline of docs/model_scoreboard.md. The 384 arm and the +curb union arm were deliberately not extended; they stay richmond-only.

Result

Pooled over the nine US cities (macro): P 0.380 / R 0.836 / F1 0.519 / AP 0.488, against RampNet's 0.953 / 0.686 / 0.793. "Transfers but does not compete" stands on every split.

The pooled number mixes two environments. richmond, clovis, morgantown, laurens_mapillary and juneau (five of the nine pooled cities), and all five held-out splits, ran on makelab2 (A40, torch 2.13.0+cu130). bend, paterson, gainesville and annapolis ran on klone (L40S, torch 2.6.0+cu126). Both used the same detection signature and transformers 5.15.0, and the two runs that record a checkpoint snapshot both loaded 4772b6bf…. The expected size of the effect is the #163 same-checkpoint environment control: the 384 arm, re-run in a different environment, landed within one detection out of 523 of the published run. No split was run in both environments for the 1024 arm.

split	RampNet P / R / F1	Vistas 1024 P / R / F1	ΔF1
clovis	0.914 / 0.713 / 0.801	0.233 / 0.877 / 0.368	−0.433
morgantown	0.975 / 0.730 / 0.835	0.379 / 0.831 / 0.521	−0.315
laurens_mapillary	0.898 / 0.390 / 0.543	0.405 / 0.787 / 0.535	−0.009
juneau	0.967 / 0.687 / 0.803	0.433 / 0.865 / 0.577	−0.226
laurens_gsv †	0.933 / 0.509 / 0.659	0.377 / 0.818 / 0.516	−0.142
budapest_district5 †	0.874 / 0.510 / 0.644	0.438 / 0.660 / 0.527	−0.118
sao_paulo †	0.894 / 0.687 / 0.777	0.340 / 0.833 / 0.483	−0.294
bayonne †	0.831 / 0.375 / 0.517	0.377 / 0.731 / 0.497	−0.020
manual_gold †	0.947 / 0.873 / 0.908	0.534 / 0.825 / 0.649	−0.260

† held out of the pool. The full fourteen-row table (with AP and tp/fp/fn) is in docs/vistas_transfer_126.md § "All fourteen splits", and a new test checks every row of it against the scorer.

Claims that changed:

• Finding 3 in the scoreboard was false once this arm pooled. It said Claude Opus 5.5 was the only model with full pooled coverage, apart from the open detectors, whose recall beats RampNet's. The Vistas arm also does, at 0.836, and it out-recalls RampNet's shipped 0.55 on all nine pooled cities. RampNet's own curve to its 0.05 floor still reaches further (micro recall 0.862 against 0.830) and is above the Vistas curve at every recall it reaches. The prose and the pinned test (test_the_newest_ids_legs_are_pooled) are updated.
• The arm is off the zero-shot manual_gold scatter. Its pooled-minus-manual_gold is −0.13, against ±0.08 for the zero-shot models, and the gain is all precision. Two candidate readings are written up; neither is measured.
• On budapest it is the strongest challenger on the board (0.527, against Opus 5.5's 0.451 and the YOLO arms' 0.221–0.277).

Where it ran, and what it cost

Run on makelab2 (one A40) on 2026-10-20, from a scratch worktree at ece4fc4 with a private --cache-dir and an offline load of checkpoint snapshot 4772b6bf…, the same snapshot the #35 klone run loaded. Before any GPU time, all nine splits' panoramas passed imagery_manifest.py --verify. juneau's panoramas were copied up from the Windows checkout, and bayonne's came from the #159 worktree on makelab2; neither is on HF. export_model_cache.py --verify gave 9/9 pairs identical to the producing cache (1,930 shards).

Each compare.py process held the A40 for its whole wall-clock, so GPU time equals wall-clock. Each process wrote one paid: false row to the run's own usage_log.jsonl, and those nine rows were then appended to analysis_out/usage_log.jsonl. The ledger's last nine lines are byte-identical to the run's file, and the 285 before them are main's ledger unchanged. The append command and a nine-row check are now written into the run instructions. makelab2 has no Slurm, so there are no compute_log.jsonl rows, per docs/compute_cost.md.

split	panos	wall-clock = GPU time (s)	detect (s)	detect s/pano
clovis	125	203.3	179.2	1.434
morgantown	125	151.5	138.4	1.107
laurens_mapillary	94	132.8	120.4	1.281
juneau	125	187.3	174.7	1.397
laurens_gsv	86	206.7	194.1	2.257
budapest_district5	125	169.3	156.9	1.255
sao_paulo	125	293.6	280.9	2.247
bayonne	125	172.7	160.3	1.282
manual_gold	1000	1191.1	1176.4	1.176
total	1930	2708.3 (0.75 A40-hours), $0	2581.3	1.337

s/pano is detect seconds over panoramas, and excludes model load (the difference between the two seconds columns). The end-to-end run took 45 min 32 s. The GPU was shared: another process held 9.2 GB at the start and the end of the run (both in run.log). Utilisation was not logged. Two nvidia-smi reads taken by hand during the run, and not recorded, showed 0% at the start and 9% during clovis, which points to CPU-side reprojection as the bound. The per-split seconds are wall-clock on a shared GPU, so they are an upper bound.

What else moved

• analysis_out/laurens_paired_151.{json,md} and docs/laurens_paired_151.md: one new leg row. No earlier row changed and the curb-probe values are identical. The recorded labeler_commit moved from 29dc605 to 39afcd4, the labeler checkout the regeneration ran from, and that field is inside rows_sha256, so the test pin moved with it.
• analysis_out/cascade_cost_35/summary.json: eight of the nine new (split, parity arm) pairs are listed under gaps as "published detections exist but this pair was not run". The ninth, laurens_gsv, has no op_cache and is not listed.
• analysis_out/eval_protocol_150/reproduction.json: the proof now covers 223 scoreboard cells, with 0 differing and 0 unchecked. This artifact is not in check_all; its CI test caught it.
• docs/replication.md (209 files, 28.2 MB; the 1024 row is now 14), the rampnet/roster.py note, docs/model_comparison.md (a coverage paragraph for the three Vistas arms, plus the generated footnotes), docs/rampnet1_findings.md (the "one split" caveat), docs/README.md and scripts/analysis/README.md.
• Regenerated scoreboard figures. In scoreboard_pr_curves.png the Vistas curve has its own colour (violet, palette slot 7) rather than the grey of the scoreless dots. The scatter figures have label nudges for the arm, and the headline figure is wider so its title is no longer clipped. These figures will conflict with figures: pooled-mean column on the by-split heatmap, rows sorted by it #262 (figures/scoreboard-heatmap-aggregate); whichever lands second should merge main and regenerate them, not merge the PNGs.

Review fixes (review 5593166337)

• The ledger append and a nine-row check are in docs/vistas_transfer_126.md (Provenance, step 3) and the vistas1024_all_splits.sh header. The command used on 2026-10-20 was not recorded. The docs say so, and say that the check verifies its result.
• The GPU-utilisation figures are labelled as unlogged hand reads.
• A mixed-environment paragraph sits under the pooled row in docs/vistas_transfer_126.md and beside the pooled number in docs/model_scoreboard.md.
• model_scoreboard.md: "apart from the two open detectors", and the curve-dominance claim now names the one exception: YOLO11x is 0.0016 higher at recall 0.087–0.090.
• Figures: the Vistas curve colour, the generalization scatter (the Vistas marker was under YOLO26's label; the Fable/YOLO11l and Gemini 3.7/YOLO26 overlaps are fixed too), the PR-curve dot labels (Gemini 3.1 Pro and Fable 5.1 no longer print over each other, and dot labels draw above the curves), and fig_headline's footer, which now reads "7 legs that cover 1–8 of the nine pooled splits", counted off the board.
• imagery_verify.log has a header with the command, the host and worktree, and the time bound. The exact time was not recorded, and the header says so.

Not run

• Complementarity, the chance null (null_recall.py) and the Experiment: hybrid RampNet + open VLM (Qwen3-VL) — ensemble / cascade / distillation #35 cascade grid on the nine new splits. All three are CPU-only re-reads of the published files, except cascade_gate.py.
• Paired intervals for the two near-ties (laurens_mapillary −0.009, bayonne −0.020). Both are point estimates.

Housekeeping

The makelab2 scratch worktree /homes/gws/jonf/wt-vistas1024 is still there: detached at ece4fc4, registered in /homes/gws/jonf/RampNet's worktree list, holding the run outputs, the tarball and a 287 MB copy of the juneau panoramas. It is pending cleanup once Jon clears makelab2 contact: git -C /homes/gws/jonf/RampNet worktree remove --force /homes/gws/jonf/wt-vistas1024. The .model_cache shards are also in D:\Git\RampNet\.model_cache.

Checks

At 0dd0b1d, on Windows (.venv, Python 3.12):

• scripts/analysis/scoreboard.py --check: all three generated outputs current.
• python -m pytest -q on the touched tests (test_scoreboard.py, test_laurens_paired_151.py, test_docs.py, test_cascade_cost_35.py, test_cascade_gate.py, test_complementarity.py, test_export_model_cache.py): 250 passed. At 806e1d6 the full suite was 3353 passed, 7 skipped.
• scripts/check_all.py --ci: 22 passed, 0 failed, 3 skipped (the slow entry and the two that need local data).
• export_model_cache.py --verify for the nine new files: 9/9 identical to the producing cache (docs/data/vistas1024_all_splits/export_verify.log).

🤖 Generated with Claude Code — Opus 5.5, claude-opus-5-5

[jonfroehlich]

Independent review (head 806e1d6): ready, after a few fixes

No blockers. Every number re-derived matches. Verified independently:

• Coverage. All 9 new files cover every pano in their split's records.jsonl: 0 missing, 0 extra, 0 uncached.
• One leg. All 14 files of the 1024 arm share one signature and published_as. The 384 and +curb arms are untouched.
• Cache. export_model_cache.py --verify against the producing cache: 9/9 identical.
• Re-scoring. An independent re-score matches every P/R/F1/AP and tp/fp/fn in all 14 doc rows, the PR body and the compare.py logs.
• Pooled. Vistas macro P 0.380 / R 0.836 / F1 0.519 / AP 0.488, against RampNet 0.953 / 0.686 / 0.793.
• Recall. Vistas out-recalls RampNet at 0.55 on all 9 pooled cities, and on 4 of 5 held-out splits (manual_gold is the exception, 0.825 vs 0.873). RampNet's 0.05-floor curve stays above the Vistas curve at every recall: micro recall 0.862 vs 0.830, minimum precision margin +0.373.
• Budapest. Vistas is the strongest challenger there: 0.527, against Opus 5.5's 0.451.
• Ledger. 9 paid: false rows, append-only (origin/main's 285 lines are a byte-identical prefix), totalling 2,708 s wall-clock.
• Bayonne. No arm masks the nadir band, so this run matches the others.
• Regeneration. check_all.py --ci: 22 passed. eval_protocol_150 223 cells, 0 differing.
• Tests. Updated honestly, none weakened.

Should-fix

1. The ledger append is missing from the run instructions (docs/vistas_transfer_126.md run section, scripts/analysis/vistas1024_all_splits.sh header). The commands write to $W/usage_log.jsonl, but the append into analysis_out/usage_log.jsonl is never written down. Add the exact command and a 9-row check.
2. The GPU-utilisation figures have no source. "0% utilisation" and "9% during clovis" appear in no committed log. Either say they were ad-hoc nvidia-smi reads that weren't logged, or drop them.
3. The mixed-environment caveat should sit beside the pooled number.
   • The 14 files come from two environments: makelab2 A40 with torch 2.13/cu130, and klone L40S with torch 2.6/cu126.
   • Both used the same signature, the same snapshot and transformers 5.15.0.
   • Name the mix in one sentence, and point to the #163 same-checkpoint env control ("within one detection out of 523") as the expected size of the effect.

Nits

• PR body.
  • It still says "Stacked on #261 … retarget to main". Both are done.
  • The cost table's 1.337 s/pano is detect-only (2581.3/1930), but there is no detect column, so it reads as inconsistent with 1.403. Label it.
• model_scoreboard.md:241. "Highest pooled recall of any model above the open detectors" can be misread as beating them. Write "apart from the two open detectors".
• model_scoreboard.md:389, a pre-existing claim this PR edited. "RampNet's curve dominates" doesn't strictly hold for YOLO11x: at recall 0.088–0.090 YOLO11x is 0.0016 higher. Soften or qualify it.
• Figures.
  • In scoreboard_pr_curves.png the Vistas curve is the same grey as the "grey dots emit no confidence" convention. Give it its own colour.
  • In scoreboard_generalization.png the thin "P" marker is hidden under YOLO26's label.
• imagery_verify.log header. It has none (command, host, timestamp). export_verify.log has one.
• Carried over from the #262 review:
  • fig_headline's footer says "single-split legs", but the partial legs include 8/9 and 5/9.
  • Now that the arm has full coverage, it has no label-nudge entry in the scatter figures.

Fixes follow, then a re-review. This PR and #262 both regenerate the scoreboard figures, so whichever merges second will merge main and regenerate.

🤖 Generated with Claude Code — Opus 5.5, claude-opus-5-5

[jonfroehlich]

Review fixes pushed in 0dd0b1d. Every finding is addressed. Checks: scoreboard.py --check current; 250 tests passed across the touched suites; check_all.py --ci 22 passed.

1. Ledger append. The run section and the script header now give the append (cat $W/usage_log.jsonl >> analysis_out/usage_log.jsonl) and a 9-row check. The check is verified against the real run file: the ledger's last 9 lines are byte-identical to it. The exact command used on 2026-10-20 was not recorded, and the doc says so.

2. GPU utilisation. The 0% and 9% figures are now described as unrecorded hand nvidia-smi reads.

3. Mixed environments. A caveat now sits beside the pooled number in both docs:

   • makelab2 A40, torch 2.13/cu130, ran richmond plus the nine new splits;
   • klone L40S, torch 2.6/cu126, ran bend, paterson, gainesville and annapolis;
   • both used the same signature and transformers 5.15.0.

   It cites the Re-run and publish the Vistas 1024-parity and A40-control detections (#126, #137) #163 control ("within one detection out of 523") and states that no split of the 1024 arm ran in both environments.

4. Nits.

   • The PR body no longer mentions stacking, and its cost table now has a labelled detect-s/pano column.
   • The scoreboard text says "apart from the two open detectors". The YOLO11x curve claim is qualified: there is a 0.0016 sliver at recall 0.087–0.090.
   • The Vistas PR curve is violet, checked against every curve it crosses.
   • New nudges fix the overlapping labels in the generalization and PR figures.
   • The fig_headline footer is counted from the board: "7 legs that cover 1–8 of the nine pooled splits".
   • imagery_verify.log has a header, and its digests match the committed manifests.

Next: a final independent re-review.

🤖 Generated with Claude Code — Opus 5.5, claude-opus-5-5

[jonfroehlich]

Final re-review (head 0dd0b1d): Ready to merge

No blockers and no should-fix items. Verified independently:

• Ledger. The documented step-3 check, run against origin/main's ledger and the run tarball, prints ok: 9 rows, 2708.3 s wall-clock, and the result is byte-identical to HEAD's ledger.
• Environments. The split-to-host assignment matches the committed logs. klone ran bend, paterson, gainesville and annapolis on an L40S (torch 2.6). makelab2 ran richmond plus the nine new splits on an A40 (torch 2.13). Both used transformers 5.15.0 and snapshot 4772b6bf.
• Curve claims. Re-derived from scoreboard.build(): the YOLO11x sliver is −0.0016 at recall 0.0875–0.0898, and the Vistas minimum margin is +0.373.
• Figures. The new colour and nudges are legible, the label overlaps are gone, and all 5 regenerated figures are pixel-identical to the committed PNGs.
• Scope. fig_by_split is untouched, so it doesn't conflict with figures: pooled-mean column on the by-split heatmap, rows sorted by it #262's code.
• Checks. scoreboard.py --check current; touched suites 250 passed; check_all.py --ci 22 passed.

Two nits applied in 6ad0dd8:

• The Re-run and publish the Vistas 1024-parity and A40-control detections (#126, #137) #163 control changed the GPU and the transformers major version. This run's mix changes the GPU and torch, with transformers fixed. Both docs now call Re-run and publish the Vistas 1024-parity and A40-control detections (#126, #137) #163 the nearest measurement rather than a control of the same change.
• Added a test for _partial_footer, whose doctest nothing ran.

Two optional nits were not applied, being cosmetic and outside this PR's claims:

• The dot-label halos cross the RampNet PR curve.
• The ΔE figures in the CURVE_COLOR comment come from an uncommitted validator.

#262 also regenerates these figures. Whichever merges second will merge main in and regenerate.

🤖 Generated with Claude Code — Opus 5.5, claude-opus-5-5

[jonfroehlich]

Merge of main (c46ff35) and its re-review: Ready to merge

main gained #259 (y11x_tiles and the seed legs) and #262 (the heatmap pooled-mean column), and was merged in at c46ff35. Generated files were regenerated with their scripts, and every count was recomputed from the merged board:

• 23 legs: 15 full and 8 partial (6 of them single-split).
• eval_protocol: 233 cells, 2,330 fields.
• 273 files, 31.3 MiB.

Prose outside the conflicts that the merge made stale is also fixed:

• claude_newest_ids.md: Opus 5.5 is now "one of two" full-coverage legs whose recall beats RampNet's, alongside Vistas 1024 (0.836).
• The Budapest YOLO range is now 0.221–0.319, with y11x_tiles at the top. Vistas 1024 is still the strongest challenger there at 0.527.

Independent re-review of the merge commit:

• Nothing is lost from either side. Each side's own changes appear exactly in the merge, file by file.
• Every count matches.
• All 5 figures re-render byte-identically.
• scoreboard.py --check and eval_protocol_150 --check pass (233/233).
• Targeted suites pass (281 tests). The full suite passed at c46ff35 (3,360). CI is green on c46ff35.

Its three findings are fixed in d746c70:

• docs/README.md said 224 cells; it now says 233.
• A missing 2026-10-20 Revision entry is added to eval_protocol_150.md.
• PEP 8 spacing restored in tests/test_scoreboard.py.

🤖 Generated with Claude Code — Opus 5.5, claude-opus-5-5