[jonfroehlich]

What

Publishes the per-panorama detections of the seven YOLO11x perspective-tiles legs, which until now existed only in makelab2's .model_cache:

• y11x_tiles, the strongest YOLO arm (Train a supervised YOLO baseline (YOLO11 / YOLO26) on the RampNet dataset — isolate architecture vs. data #51, docs/yolo_geometry_51.md), ten splits
• the six Run B (#84): the gate passed — decide whether to spend 1,675-3,350 GPU-hours on the annealed arm #135 seed-variance legs y11x_tiles_s{1,2}_ep44, y11x_tiles_s3_ep42, y11x_tiles_s{1,2,3}_best, nine splits (no budapest_district5)

64 files, 3.1 MB, under benchmark/model_detections/. Found while checking the inputs for #258's YOLO rung; audit in the #51 comments.

How it was checked

• Exported, not re-inferred. export_model_cache.py on makelab2 against the cache that scored the published numbers (A40, ultralytics 8.4.120). The cache key includes a SHA-1 of the weights file, so each file holds the detections of the exact checkpoint that was scored.
• --verify: 64 pairs score identically to the cache, 0 uncached.
• A compare.py re-score of bend from the same cache, in a clean checkout of main, matches every row of docs/data/seed_variance_51_135/yolo/bend_tiles.txt.
• The checkpoints on makelab2 match the sha256s in env.txt. So do the copies on klone, including the three _ep checkpoints recovered from Tillicum today.
• Full suite: 3,216 passed, 7 skipped, plus the two re-fixed test files afterwards (Windows venv).

What else changed, and why

Everything that reads the roster had to be regenerated:

• rampnet/roster.py: seven standing=False entries, like the pano trio. Not in SCORED_SPECS.
• Scoreboard: JSON, docs/model_scoreboard.md, docs/model_comparison.md tables, two figures, display names. Only y11x_tiles is on the board, in the partial table (7/9: no laurens_mapillary or juneau), so no pooled number moves. The six seed legs are published and registered but kept off the board by a new roster field, seed_of; scoreboard.json lists them under off_scoreboard.
• analysis_out/eval_protocol_150/reproduction.json: 224/224 scoreboard cells equal (23 legs, after the merges of Juneau (#254): reviewed, scored, registered as the ninth pooled US split #260 and Newest Claude ids on every split: Opus 5.5 and Fable 5.1 #261).
• analysis_out/cascade_cost_35/summary.json: only gains "published but not run" rows.
• docs/replication.md: file counts (264 files, 30.6 MiB after the merges), a ledger row, the exact export commands, and where the checkpoints are.

One finding changes. model_scoreboard.md said Claude Fable 5 is the strongest challenger on annapolis (F1 0.611). With the tiles arm on the board it isn't: YOLO11x tiles is 0.640 (0.636–0.655 across seeds). Since #261, Claude Opus 5.5 (0.720) tops both. The sentence is rewritten to say so. It still describes Fable as the best zero-shot leg there. The pooled-table and pano-arm claims elsewhere are scoped tightly enough that they still hold; I checked.

Decided

• The six seed legs stay off the scoreboard (6784f05). They are published, registered and re-scorable; seed_of keeps one recipe from appearing seven times.
• Checkpoints are not published. They are on klone and makelab2 with matching hashes. Nothing here needs them, because re-scoring reads the detections.

Review

Round 1 review → fixes → round 2 re-review → fixes are in the comments (2026-10-19, through 137e34b). The later changes (6784f05, c25bf65 and the two merges of main) had an independent re-review on 2026-10-20: ready to merge, with its fixes in 600d497.

🤖 Generated with Claude Code — Opus 5.5, claude-opus-5-5

[jonfroehlich]

Independent review, round 1 (head b66e0cf)

Verdict: the data is correct. No blockers. One Major and four Minor findings, all of them doc prose that this PR makes stale. I made no changes to the branch.

What I verified (no defects)

1. Published files: none. I checked all 64 benchmark/model_detections/y11x_tiles*.json with a script:

• model, published_as and signature.model_id all equal the leg label. city equals the split in the filename.
• Every signature has conf 0.05, imgsz 1024, tile: true and provider yolo, plus the six-view perspective layout. All 64 files share one signature apart from the id and hash fields.
• n_uncached is 0 everywhere, and n_panos equals the number of detection keys.
• Each file's pano set equals its bundle's records.jsonl pano set exactly: 125, 124 for richmond, 110 for bend, and 1,000 for manual_gold.
• Every file is byte-identical to json.dumps(d, **export_model_cache.DUMP_KW), so all are canonical.
• There are seven distinct weights_hash values, one per leg. I computed sha1sum on klone's copies of the checkpoints and all seven match: y11x_tiles/weights/best.pt = 05e2edd2…, seedvar/y11x_tiles_s1/weights/epoch43.pt = 19cf044f…, and so on. The sha256s of the same files match env.txt in both docs/data/ directories. So each file holds the detections of the checkpoint that was scored. I checked this directly, not by relying on the cache key.

Scoreboard against the committed reports: all 64 (leg, split) cells match. I parsed every y11x_tiles* row of docs/data/yolo_geometry_51/*_tiles.txt (seed 0) and docs/data/seed_variance_51_135/yolo/*_tiles.txt (seed legs). I compared them with analysis_out/scoreboard.json per_split on tp/fp/fn, P/R/F1 and AP, all at conf 0.25. Result: checked 64 bad 0. The scoreboard has no tiles cell without a report row, and no report row is missing from the scoreboard.

2. Roster: none. The seven entries are all standing=False and none is in SCORED_SPECS. "Ten splits" for seed 0 and "Nine splits, no budapest_district5" for the seed legs match the file inventory (10, 9, 9, 9, 9, 9, 9). The prefix 63bbe5d6 matches docs/data/yolo_geometry_51/env.txt. The note that epochN.pt is results.csv row N+1 matches run_seedvar_eval.sh and check_epoch_ckpt.py. On klone, epoch43.pt/epoch43.pt/epoch41.pt hash to the _ep44/_ep44/_ep42 sha256s in env.txt.

docs/replication.md counts and paths: none.

• git ls-files gives 226 detection JSONs outside replicates/, totalling 30,034,747 B, which is 28.6 MiB (same convention as the old 25.6). The 64 new files are 3,237,882 B, the 3.1 MB in the PR body.
• The table rows sum to 226.
• The klone checkpoint paths exist as written.
• The export commands are consistent with export_model_cache.py: legs with no cache on a split are skipped, not written.
• I could not check the makelab2-side claims (the seedvar_ckpts/SHA256SUMS staleness and the mislabelled/ directory) because I did not connect to makelab2.

4. Regenerated artifacts and tests: none.

• scoreboard.py --check reports all three outputs current.
• eval_protocol_150.py --check gives scoreboard 239/239 and benchmark_eval 30/30 equal. reproduction.json cells went from 205 to 269, which is +64.
• cascade_cost_35/summary.json gaps went from 10 to 74, which is +64, all of them "published detections exist but this pair was not run".
• Targeted tests: 407 passed and 1 skipped in scoreboard/roster/cascade/eval_protocol/export/complementarity/benchmark_eval/claude_legs/bayonne; 397 passed in docs/docs_index/model_comparison/parity/seed_variance/yolo_geometry/multiview/silent_witness/fp_taxonomy/epoch_ckpt.
• The test edits are honest updates, not weakening: the coverage entries are added, _WORDS is extended so "Fifteen" can be parsed, and the stale-prose test is re-pointed to the new "Twenty-eight" sentence.

5. PR body: none false. The reworded annapolis bullet is accurate. YOLO11x (tiles) scores 0.640, the seed legs 0.636–0.655, and RampNet 0.839. Fable 5 (0.611) is the best zero-shot leg on annapolis. "Fifteen legs … seven of them only one" also checks out (3 Vistas + 5 Claude + 7 tiles = 15; the 2 richmond-only Vistas legs plus the 5 annapolis-only Claude legs make 7). The new Omissions bullet's split list matches the files. I did not re-run the full suite, so the claim of 3,216 passed is unverified by me.

Findings

[Major] docs/operating_point_parity_51.md:108-110 and :154-160 still say the tiles detections are unpublished and the arm is unregistered.

• Line 109: "It cannot be run from a clean clone, because the tiles arm's detections are not published (see Gaps)."
• Lines 154-160: "The tiles arm's detections are not published … y11x_tiles and y11x_pano are not in rampnet/roster.py and have no files under benchmark/model_detections/ … Registering them as roster legs and exporting their detections is what unblocks it."

This PR does exactly that for y11x_tiles, so the Gaps entry is now false for half its subject. It is also the doc the scoreboard cites for "at matched operating points y11x_tiles leads RampNet". Evidence: grep -n "detections are not\|not in \rampnet/roster.py`" docs/operating_point_parity_51.mdgives lines 109, 154 and 156. Neither file is ingit diff --stat origin/main...HEAD. *Fix:* say the tiles detections are published (#258) and the arm is registered. Keep y11x_pano` as the remaining unpublished arm. Reword the blocker for the paired test: it is now "not run", not "cannot be run".

[Minor] docs/yolo_geometry_51.md:200-202: "Publishing the three checkpoints, or their per-pano detections, is what would change that; neither is done." The tiles arm's detections are now published. y11x_pano_h200's were already published as part of the pano trio. Only y11x_pano is still cluster-only.
Fix: add a line saying y11x_tiles (and y11x_pano_h200) now re-score from a clean clone, and that y11x_pano does not.

[Minor] docs/rampnet1_report.md:630: "Several inputs live only on cluster storage: the YOLO checkpoints and the tiles arm's detections, …" This is no longer true of the detections. If the 1.0 report is meant to be a frozen snapshot, a dated "since published (#258)" note is enough. Otherwise drop "the tiles arm's detections".

[Minor] docs/eval_protocol_150.md:9-11 and :128 still quote the pre-PR proof. They say "all 175 cells of analysis_out/scoreboard.json (21 model legs, 13 splits) … 1,750 … field comparisons", and the results table row is 175 | 175 | 0 | 0 | 1,750. The regenerated reproduction.json that this PR commits has 239 scoreboard cells and 2,390 fields over 28 legs:

$ python scripts/analysis/eval_protocol_150.py --check
scoreboard          239    239        0          0    2390
benchmark_eval       30     30        0          0     480

The doc's own Revision section shows the convention: the Bayonne change got a dated entry ("175 scoreboard cells (was 165…)").
Fix: update the counts and add a dated revision line for the 64 new cells.

[Minor] docs/model_scoreboard.md:327-329, the budapest bullet: "The three YOLO arms fall to 0.221–0.277, below all three Gemini legs (0.336–0.381)." The matrix directly above now has a fourth YOLO row on budapest, YOLO11x (tiles) at 0.319. The conclusion still holds, since 0.319 is below 0.336, but the count and range no longer describe the table.
Fix: "The four YOLO arms that ran it fall to 0.221–0.319 …".

[Nit] docs/model_scoreboard.md:75: "(Claude Fable 5 at low effort scores 0.611, but on one split only; see the partial table below.)" This parenthetical points readers to the partial leg that could be mistaken for beating YOLO11l's pooled 0.599. The tiles legs now do that more strongly: macro F1 over 7/8 pooled splits is 0.667 for seed 0, and every tiles leg is above 0.599 on most splits. The sentence is still true, but the likelier misreading now goes unflagged.
Fix: add "the YOLO11x tiles legs reach ~0.67 over seven of the eight pooled splits (no laurens_mapillary)".

[Nit] docs/cascade_cost_35.md:220, the heading "## (d) Every published challenger on richmond". This PR publishes seven more richmond challengers that section (d) did not run. summary.json now records them under gaps, but the heading reads as if it were complete.
Fix: say "as of 2026-09-xx" or "every challenger published at the time", or point to the gaps list.

Categories with no defects: published-file correctness, scoreboard-vs-report numbers, roster entries, replication.md counts/commands/klone paths, regeneration consistency, test honesty, PR-body claims.

🤖 Generated with Claude Code — Opus 5.5, claude-opus-5-5

[jonfroehlich]

Round-1 review fixes (commit c7a60f0)

All seven findings are fixed, each one doc prose only. I re-derived every number from analysis_out/scoreboard.json, analysis_out/eval_protocol_150/reproduction.json (compared against origin/main's copy) and analysis_out/cascade_cost_35/summary.json. None comes from the review text.

1. [Major] docs/operating_point_parity_51.md:108-112 and :156-165. Both now say the tiles detections are published and y11x_tiles is registered (Frontier model as a verifier on RampNet's sub-threshold candidates: recover recall with few-shot crops (RampNet 2.0, hybrid) #258, 2026-10-19). The paired test now reads "has not been run", not "cannot be run". I checked what it needs: RampNet's analysis_out/op_cache/{bend,morgantown,manual_gold}.json is committed, and so is benchmark/model_detections/y11x_tiles__<split>.json for the same splits. Both are floored at 0.05, which is also the tiles arm's selected parity threshold. y11x_pano is still named as neither registered nor published. Both arms' checkpoints are still cluster-only.
2. [Minor] docs/yolo_geometry_51.md:200-205. The checkpoints are still unpublished. Per-pano detections are now published for y11x_pano_h200 (with the pano trio) and for y11x_tiles (Frontier model as a verifier on RampNet's sub-threshold candidates: recover recall with few-shot crops (RampNet 2.0, hybrid) #258), so those two arms re-score from a clean clone. y11x_pano still does not.
3. [Minor] docs/rampnet1_report.md:630-633. The claim itself is unchanged. I added a dated note at the end: "(Since published: the tiles arm's detections, 2026-10-19, Frontier model as a verifier on RampNet's sub-threshold candidates: recover recall with few-shot crops (RampNet 2.0, hybrid) #258.)"
4. [Minor] docs/eval_protocol_150.md.
   • :4-5: the header now mentions the 2026-10-19 addition.
   • :9-11: now 239 cells, 28 model legs, 13 splits, and 2,390 fields.
   • :128: the table row is now 239 | 239 | 0 | 0 | 2,390.
   • :257-264: a new dated revision entry. It records that all 64 new cells are y11x_tiles* and that no old cell was dropped. The 11 op_cache cells, the split pins and the scorer fingerprint f4aef67aba03 are unchanged.
5. [Minor] docs/model_scoreboard.md:330. The text now reads "The four YOLO arms that ran it fall to 0.221–0.319". The budapest values are y11x_pano_h200 0.221, y11l_pano 0.247, y26_pano 0.277 and y11x_tiles 0.319. The conclusion still holds, because 0.319 is below the lowest Gemini leg (gemini-3.6-flash, 0.336).
6. [Nit] docs/model_scoreboard.md:75-78. The parenthetical now also says that the seven tiles legs reach 0.666–0.678 macro F1, but over 7/8 pooled splits (no laurens_mapillary). It adds that neither partial figure is comparable to a full-coverage mean. This is prose only; the generated tables are untouched.
7. [Nit] docs/cascade_cost_35.md:220-225. The heading is now "Every challenger published at the time". A new paragraph says the 15 challengers in the table are the ones that existed when the section was run. It also says the seven tiles legs were not run in (d) or (e), and that summary.json lists all 64 of their (leg, split) pairs under gaps.

Checks

• scripts/analysis/scoreboard.py --check: all three outputs current.
• scripts/analysis/eval_protocol_150.py --check: scoreboard 239/239 equal over 2,390 fields; benchmark_eval 30/30 equal over 480 fields.
• pytest -q tests/test_docs.py tests/test_scoreboard.py tests/test_eval_protocol_150.py tests/test_cascade_cost_35.py tests/test_roster.py tests/test_export_model_cache.py: 240 passed.

🤖 Generated with Claude Code — Opus 5.5, claude-opus-5-5

[jonfroehlich]

Independent re-review, round 2 (head c7a60f0)

Verdict: the seven round-1 fixes are correct and complete, and they introduced no new errors. Round 1 missed two sentences that this PR makes false: one Major, one Minor. Both are one-line prose fixes. Fix them, then merge. No re-review needed after that. I made no changes to the branch.

Round-1 findings: all verified fixed

I re-derived every number in c7a60f0 from committed artifacts. None of them came from the fix comment.

1. [Major] operating_point_parity_51.md: fixed.
   • The three near-tied splits in §2 are bend, morgantown and manual_gold.
   • For each of them, analysis_out/op_cache/<split>.json (meta.score_floor 0.05) and benchmark/model_detections/y11x_tiles__<split>.json (signature.conf 0.05) are both committed. Their pano sets are identical: 110/110, 125/125 and 1000/1000.
   • So "the test can now be run from a clean clone" is true.
   • y11x_pano has no detection files, so "still neither registered nor published" is also correct.
2. [Minor] yolo_geometry_51.md:202-206: fixed.
   • y11x_pano_h200__*.json is committed for 13 splits.
   • Its scoreboard bend cell (181/2/146, F1 0.710, AP 0.781) matches the control row in docs/data/yolo_geometry_51/bend_pano.txt.
   • The roster spec points to the same yolo_ckpts/y11x_pano_h200.pt that env.txt hashes.
3. [Minor] rampnet1_report.md: annotation only. The diff adds a single dated parenthetical, and every original claim is byte-unchanged. That is correct for a frozen record.
4. [Minor] eval_protocol_150.md: fixed.
   • reproduction.json on this branch has 239 scoreboard cells, 239 equal, 2,390 fields, over 28 legs and 13 splits. origin/main has 175 cells and 1,750 fields.
   • I diffed the cell keys: there are 64 new cells, all y11x_tiles* (10 + 6×9), all in the scoreboard source. No cell was dropped and none of the 205 existing cells changed.
   • The op_cache cell count is 11 on both, the split pins are equal, and the fingerprint is f4aef67aba03.
   • The revision entry matches all of this.
5. [Minor] model_scoreboard.md:330: fixed. Budapest F1 is y11x_pano_h200 0.221, y11l_pano 0.247, y26_pano 0.277 and y11x_tiles 0.319. The Gemini legs are 0.336, 0.338 and 0.381, so the four YOLO arms are below all three.
6. [Nit] model_scoreboard.md:75-78: fixed. Macro F1 of the seven tiles legs over the seven pooled splits they ran ranges from 0.665528 to 0.677504, which rounds to 0.666–0.678. Every leg is missing exactly laurens_mapillary, and coverage is 7/8.
7. [Nit] cascade_cost_35.md:220-225: fixed. Table (d) has 15 rows. summary.json gaps has 74 entries, and 64 of them are y11x_tiles* with reason "published detections exist but this pair was not run".

I found no broken links or contradictions in the fix commit. It touches no shell blocks, so no line continuations could have collapsed. The #258 links resolve: that issue's rung 4 is where the gap was found.

New findings (pre-existing text that this PR makes false)

[Major] docs/model_scoreboard.md:541-547. This is hand-written prose between the ap-provenance and coverage generated blocks. It says:

The YOLO tiles arms are absent from this board … It is absent here because its detections are not published and it is not in rampnet/roster.py, not because it is untested.

After this PR, all seven tiles legs are on this board: in the partial table (from line 160) and in the by-split matrix. All three parts of the sentence are now false, and the doc contradicts itself. It is the same kind of defect as round 1's Major. This text was at origin/main:docs/model_scoreboard.md:470.

Fix: replace the bullet with: "The YOLO tiles arms are in the partial table only. y11x_tiles and the six seed legs were published and registered on 2026-10-19 (#258). None of them ran laurens_mapillary, so they have no pooled number." Keep the pointer to yolo_geometry_51.md.

[Minor] docs/model_comparison.md:275-277. This is hand-written prose (it was at origin/main:269). It says:

The tiles arms … are absent from every table above because their detections are not published, not because they are unmeasured.

They are still absent from the tables above. The roster table at lines 78-84 says "published, not in these tables". But the stated reason is now false, and the per-split "Also scored on this split" lines further down (361, 380, 402, 438, …) now list every tiles leg.

Fix: change the reason to: "absent from the standing tables above because they are not standing legs, and published since 2026-10-19 (#258). They appear in each split's 'Also scored' line below."

I swept the rest of docs/ for y11x_tiles / "tiles arm" combined with published / clean clone / cluster / roster. The only other hits are operating_point_parity_51.md:157 and rampnet1_report.md:630, and both are correctly dated history.

Checks

• scripts/analysis/scoreboard.py --check: all three outputs current.
• scripts/analysis/eval_protocol_150.py --check:
  • scoreboard: 239/239 equal over 2,390 fields, 0 differing, 0 unchecked.
  • benchmark_eval: 30/30 equal over 480 fields.
  • Fingerprint f4aef67aba03.
• pytest -q tests/test_docs.py tests/test_scoreboard.py tests/test_eval_protocol_150.py tests/test_cascade_cost_35.py tests/test_roster.py tests/test_export_model_cache.py: 240 passed.
• CI (gh pr checks 259): checks, pytest (3.10) and pytest (3.12) all pass.

🤖 Generated with Claude Code — Opus 5.5, claude-opus-5-5

[jonfroehlich]

Re-review findings fixed in 137e34b:

• [Major] docs/model_scoreboard.md (the tiles-arms bullet): the bullet now says the seven tiles legs were published and registered on 2026-10-19 (Frontier model as a verifier on RampNet's sub-threshold candidates: recover recall with few-shot crops (RampNet 2.0, hybrid) #258), and sit in the partial table because none ran laurens_mapillary. It no longer says they are absent for lack of published detections.
• [Minor] docs/model_comparison.md (~275): the tiles arms are absent from the standing tables because they are not standing legs. Their detections have been published since 2026-10-19, and they are listed in the per-split "Also scored" lines.

scoreboard.py --check current; tests/test_docs.py + tests/test_scoreboard.py 101 passed. The re-review said these two fixes need no further review round.

🤖 Generated with Claude Code — Opus 5.5, claude-opus-5-5

[jonfroehlich]

Independent re-review, round 3 (head 24069f4, covering 137e34b..24069f4): Ready to merge

This round covers everything since the last reviewed change. That is the seed-leg change (6784f05), its fixes (c25bf65), and the merges of #260 (704c58f) and #261 (24069f4). The authoring session had ended, so the review ran from another session.

Verified

• seed_of mechanism: correct and applied consistently. 23 legs on the board equals 29 roster entries minus the 6 seed legs. unregistered_exports() is [].
• Consumers:
  • eval_protocol_150 (224/224 cells) and cascade_cost_35 (64 tiles gaps listed) are consistent.
  • laurens_paired_151 is unaffected.
  • The export-file tests still cover the seed files.
• Seed files: all 54 re-score to exactly the numbers in docs/data/seed_variance_51_135/yolo/*_tiles.txt, with 0 differing.
• Both merges kept both sides (checked with git show --remerge-diff):
  • Juneau stays pooled, and y11x_tiles is re-scoped to 7/9.
  • From Newest Claude ids on every split: Opus 5.5 and Fable 5.1 #261: the Opus 5.5 headline, the 9/9 coverage, the paired-lead prose and the claude_newest_ids_261 check are all kept.
• Counts match the board: 23 legs and 14 splits; 9 partial legs, 6 of them single-split; 264 files and 30.6 MiB.
• Tests: none weakened against origin/main. A new test_seed_legs_are_named_but_not_scored was added.
• Checks:
  • scoreboard.py --check, eval_protocol_150 --check and check_all.py --ci (22 passed) all pass.
  • The full pytest run fails only the 43 known Windows-only slurm tests.
  • CI is green.

Fixed in 600d497, plus a PR-body update

• The PR body was stale after the seed-leg change and the merges: "all seven legs in the partial table", 239 cells, 226 files, and the open seed-leg question. It now records the outcome.
• eval_protocol_150.md's Result paragraph had been resolved to main's 187 / 21 legs. It now gives 224 / 23 / 2,240.
• replication.md's scoreboard check row now excludes the seed legs.
• roster.py no longer miscounts its property list.
• A new roster test requires seed_of to name a registered, non-seed parent, and requires every seed leg to be non-standing.

Not done (optional)

• There is no committed test that re-scores the 54 seed files; they were verified by hand here.
• The talk README's "18 annapolis challengers" was already stale on main. The talk is historical.

🤖 Generated with Claude Code — Opus 5.5, claude-opus-5-5