[jonfroehlich]

Steps 1 and 2 of the #258 verifier plan (#258 (comment)), plus the leakage check. Write-up: docs/verifier_258.md.

Result: the production CurbRamp validator ranks RampNet's candidates worse than RampNet's own score does on 13 of 14 splits, so the design's stop condition is met. Below the 0.30 operating point it ranks better on the pooled verdict splits, but not on manual_gold. Draft because the one follow-up (a combined rule, below) needs Jon's call.

Step 2: the validator on every candidate

scripts/analysis/validator_score_258.py (GPU) cuts the crop production would cut, following the sidewalk-ai-api 745e0c5 serving path with the perspective function copied verbatim:

• 8192×4096 resize;
• 90° 2048² perspective;
• Depth-Anything-V2 metric vitb/vkitti, with the checkpoint's sha256 asserted against serving's pin;
• crop half-size int(6100/depth), resized to 640;
• validator 969ad5a3…, recording P(correct).

It resumes per pano. validator_score_258.slurm checks every split's imagery_manifest.json before scoring.

scripts/analysis/validator_auc_258.py (CPU, --check) computes the AUC of P(correct) against RampNet's score, with a paired pano bootstrap, plus the untuned two-pass rule.

Run: klone job 42152480, one L40S, 0.51 GPU-h, $0, no preemption. The compute_log.jsonl row, the sacct dump, the job log and all 14 score files are committed.

	RampNet score	validator P(correct)	validator − RampNet
open pool, all candidates	0.878	0.823	−0.054 [−0.077, −0.033]
reference pool, all candidates	0.946	0.872	−0.074 [−0.091, −0.060]
manual_gold, all candidates	0.953	0.849	−0.104 [−0.119, −0.090]
laurens_mapillary, all candidates	0.847	0.902	+0.055 [0.008, 0.102]
open pool, below 0.30	0.698	0.768	+0.070 [0.024, 0.117]
reference pool, below 0.30	0.689	0.782	+0.092 [0.036, 0.148]
manual_gold, below 0.30	0.709	0.705	−0.004 [−0.071, 0.065]

• No variant changes a sign. Dropping the redundant FPs or restricting to the unbiased subset shifts no split's sign.
• The below-0.30 stratum was added after richmond only (b8287cc), before any other split's scores were read. It was not in the posted design.
• Untuned two-pass rule (floor 0.05, then the validator's own argmax): F1 falls on 12 of 14 splits. It rises on laurens_mapillary, 0.660 → 0.739 (recall 0.53 → 0.66), and is flat on laurens_gsv.
• The validator calls most candidates "correct" (80% on richmond, 87% on manual_gold), so its 0.5 cut is a weak filter.

Instrument differences, recorded in the doc:

• manual_gold's panos are 4096 wide, so its crops are upsampled 2× (native panos are Publish deployment validation ground-truth as a HuggingFace dataset (Bend GSV + Richmond Mapillary) #21). This is the open question behind its below-0.30 row.
• transformers is 5.15.1 against serving's 5.20.0.
• xFormers was absent.

Proposed, not run: keep RampNet ≥ 0.30 and admit a sub-threshold candidate only above a high P(correct) bar, with the bar fitted leave-one-split-out.

Step 1 and the leakage check (unchanged)

• verifier_candidates_258.py labels all 10,504 RampNet peaks ≥ 0.05 on 14 splits, once each. The label holds at every threshold, and the script asserts this against score_pano. Recall ceiling: laurens_mapillary 0.78, bayonne 0.72.
• validator_leakage_258.py finds 0 pano-id overlaps and 0 benchmark panos within 500 m of any CurbRamp label in the validator's 19 training cities, on all 14 splits. The nearest is 2.3 km away.

Tests

tests/test_validator_auc_258.py (new, 12) and tests/test_verifier_candidates_258.py pass, as does tests/test_docs_index.py. validator_auc_258.py --check and verifier_candidates_258.py --check are current. The full suite is left to CI.

🤖 Generated with Claude Code — Opus 5.5, claude-opus-5-5

[jonfroehlich]

Review of the step 2 read-out (docs/verifier_258.md, validator_auc_258.py)

The scoring run, the leakage bound and the candidate set are solid and well pinned. The conclusion is drawn on the wrong stratum, and once it is read against the design actually posted on #258 the verdict flips from "stop" to "step 3 as planned". Findings in severity order.

1. The primary AUC answers a question the design did not ask, and the "stop rule met" rests on it

The posted design (#258, comment 6316627866, step 1) defines the candidate set as RampNet peaks with score in [0.05, 0.55), and step 2's stop rule is the AUC on that set. Step 1 here widened the set to every peak ≥ 0.05 ("a two-pass pipeline filters everything above its lowered threshold"). That is a defensible pipeline choice, but it changes the primary metric: over the full range RampNet's own score is being asked to separate its confident TPs from its 0.05-floor FPs, a separation it wins by construction because the candidate set was selected by that same score (incorporation bias: the ranker under test defined the sample). The 13-of-14 "worse" result is mostly that.

The below_op table ([0.05, 0.30), the band a lowered threshold actually adds) is the closest thing in the PR to the pre-registered band, and there the pre-registered stop rule is not met: open pooled +0.070 [0.024, 0.117], reference pooled +0.092 [0.036, 0.148]. The doc labels below_op as the post-hoc deviation and the all-peaks AUC as primary. It is the other way round. Please:

• State in the Step 1 section that widening to all peaks departs from the posted step 1, and why.
• Add the literal pre-registered band, [0.05, 0.55), as a third stratum (it is a one-line change to below_op; keep the 0.30 one too since the operating point moved to 0.30 after the plan's wording).
• Make the pre-registered band the primary row of the Result paragraph, and move the all-peaks AUC to a secondary table with the selection caveat beside it.
• Rewrite the headline: the stop condition is not met on the open pool; it is met on manual_gold and not met on laurens_mapillary; bayonne is flat.

2. Multiplicity around the post-hoc stratum is handled, but say what richmond showed when it was added

The disclosure is good. Add the sentence that makes it checkable: richmond's below_op difference was +0.084 [−0.039, 0.222] (not significant) at commit b8287cc, so the stratum was not added because the first split looked favourable. With 14 splits × 3 variants × 2 subsets, say explicitly that the pooled rows are the inferential ones and the per-split rows are descriptive.

3. The "untuned two-pass rule" is not the rule the design proposed, and the doc reads it as evidence

Step 3 of the posted design says: fuse the two scores, fit on sao_paulo only, report recall at matched precision against the threshold-only control in cascade_cost_35.py. The table here instead hard-gates at the validator's production cut (P > 0.5), which the doc itself notes passes 80–87% of candidates, then concludes the rule "lowers F1 on 12 of 14 splits". That is a straw man and should not sit under a "Secondary" heading as if it were a pre-planned test. Either run step 3 as written (sao_paulo dev, everything else held out; the design already fixed the dev split, so leave-one-split-out is not needed) or move this table to an appendix titled "the production cut as a filter, for reference".

4. The manual_gold 2× upsample confound has a cheap direct test

Both readings of the manual_gold row are left open pending #21. They can be separated now: downsample one GSV verdict split (richmond, native 8192 panos on klone) to 4096, re-run the same .slurm, and compare richmond's below_op AUC native vs upsampled. One L40S job, well under the 30 min the full run took, $0. If upsampling costs richmond the same ~0.08 that separates manual_gold from the verdict pool, reading 2 wins and the manual_gold row stops being the counter-evidence.

5. "Open pool" contains a GSV split

The open pool is taken from open_rig_misses_258.py and includes laurens_gsv, which is GSV. For a rig-transfer question the open pool should be the four non-GSV splits (laurens_mapillary, bayonne, budapest_district5, juneau), with laurens_gsv reported beside laurens_mapillary as the same-town control, as #265 does. Recompute the pooled rows that way; keep the current definition only if it is needed to match #265's tables, and then say so.

6. The gain is laurens_mapillary, not "open rigs"

Within the GoPro Max pair, laurens_mapillary is +0.218 [0.135, 0.309] below the operating point and bayonne is −0.005 [−0.094, 0.082]. #35 also gained only on laurens_mapillary. The write-up should say the evidence is one town, flag that bayonne (the other GoPro Max split, Panoramax) does not show it, and name the candidate explanations (Laurens is where RampNet's own ranking is weakest, 0.847, and where sub-threshold misses dominate per #265).

7. Ledger: the compute row exists twice

The doc says the 42152480 row is in analysis_out/compute_log.jsonl. It is committed in this branch, and the same row is also sitting uncommitted in the main checkout's working tree (D:\Git\RampNet on cluster-review-224), which is where the ledger appends by default. Whoever commits that checkout will duplicate it. Add a note in the PR body that the main-checkout copy is to be discarded, or drop it now.

8. Smaller points

• The AUC bootstrap resamples panos but the two-pass F1 cells carry no interval; add the same paired bootstrap to the F1 differences if the table stays.
• "Status" line and the Result paragraph both say "stops here"; both need the rewrite from finding 1.
• The design's step 2 said "if it does not beat RampNet's score on the open-rig splits, stop". Record in the doc that the run scored all 14 splits rather than the five named, and that nothing was fitted on them, so the held-out splits remain usable for step 3.
• Step 1 caveat "TP labels on sub-threshold peaks are a lower bound" is right; add the consequence for the AUC comparison in one line: an unmarked real ramp is scored high by the validator and low by RampNet, so the bias penalises the validator, which makes the below-op gain conservative.

What I would merge

After 1, 3 and 5 are addressed the PR is a positive result for the pre-registered band on the open pool, driven by one split, with step 3 (fusion fitted on sao_paulo, recall at matched precision) as the committed next step and the richmond downsample check as the cheap confound test. That is a different document from the one here, so this is request-changes in substance even though I am posting it as a comment on a draft that another session owns.

🤖 Generated with Claude Code — Fable 5.1, claude-fable-5-1