[jonfroehlich]

Addresses the review on #269 (findings 1 to 8) and fixes #269's failing CI. Base is analysis/validator-verifier-258, so this merges into #269, not main.

The headline changes, and not quite the way the review predicted

Read on the pre-registered band, RampNet score in [0.05, 0.55), the validator does not beat RampNet's score on the open pool: +0.017 [−0.024, 0.059] AUC. The review expected the stop condition to be unmet there. It is met on the open pool and on manual_gold (−0.074 [−0.117, −0.031]), and not met on laurens_mapillary (+0.129 [0.066, 0.198]). bayonne is flat (−0.023). The post-hoc sub-0.30 stratum still shows the gain (+0.076 [0.027, 0.129] on the four-split open pool). The doc says this plainly.

Step 3, run as posted (finding 3)

scripts/analysis/validator_fusion_258.py (CPU, --check, tests/test_validator_fusion_258.py). It fits a logistic regression on logit(RampNet score) and logit(P(correct)) on sao_paulo only, and holds out every other split. Fitted weights: 1.18 for RampNet and 0.20 for the validator. Kept sets are re-scored exactly with score_pano. "Matched precision" is cascade_cost_35's best_recall_at_precision_ge_baseline definition applied to both arms, with a selection-aware paired pano bootstrap.

	threshold-only R	fused R	fused − threshold-only
open pool (4 non-GSV)	0.610	0.635	+0.024 [+0.003, +0.054]
GoPro Max pool	0.511	0.607	+0.096 [+0.043, +0.144]
reference pool	0.793	0.805	+0.012 [+0.001, +0.023]
all 13 held-out splits	0.813	0.821	+0.007 [+0.000, +0.013]
laurens_mapillary	0.526	0.659	+0.133 [+0.035, +0.193]
bayonne	0.498	0.578	+0.080 [+0.003, +0.127]
manual_gold	0.892	0.893	+0.001 [−0.005, +0.006]

With the fused cut also fixed on sao_paulo (0.285) and applied unchanged, the open pool gains +0.020 [+0.007, +0.034] F1 at unchanged precision. The reference pool and manual_gold come out flat. The untuned P > 0.5 table is now an appendix, "the production cut as a filter, for reference", exactly re-scored and with bootstrap ΔF1 intervals.

The other findings

• 1: a prereg_band stratum is added beside below_op, for all and unbiased, per split and pooled. The Result and Status lines now read on that band. All-peaks is a secondary table with the selection-bias caveat. Step 1 says why it widened the posted set.

• 2: the richmond below_op difference at b8287cc (+0.084 [−0.039, 0.222]) is stated. Pooled rows are inferential, per-split rows descriptive.

• 4, the richmond 4096 control: klone job 42153716 (one L40, 0.12 GPU-h, $0). Richmond's native panos were downsampled with manual_gold's own transform (cv2.resize INTER_LINEAR to 4096×2048, then a default cv2.imwrite). Validator AUC, native vs 4096:

  • band: 0.740 vs 0.723 (−0.017 [−0.041, 0.004])
  • below 0.30: 0.751 vs 0.754 (+0.003 [−0.025, 0.028])
  • all peaks: 0.860 vs 0.855 (−0.005 [−0.011, 0.000])

  Upsampling does not explain manual_gold's flat row. The scores, downsample manifest, job log, ledger row and sacct dump are committed.

• 5: the open pool is now the four non-GSV splits, with laurens_gsv shown beside laurens_mapillary as the same-town control. The old five-split pool stays in the JSON as open_with_laurens_gsv. Open-rig miss buckets: miss_taxonomy --cities + open_rig_misses_258 (#263 Track 1) #265's tables pool only the GoPro Max pair, so parity with Open-rig miss buckets: miss_taxonomy --cities + open_rig_misses_258 (#263 Track 1) #265 does not need the old pool.

• 6: the doc now says the gain is laurens_mapillary, one town, with bayonne flat in AUC, and names the candidate explanations. In step 3, bayonne does gain at matched precision.

• 8: bootstrap intervals are on every kept F1 difference. The "all 14 scored, nothing fitted" note is added, and so is the one-line consequence of the TP lower bound.

CI fix

#269's test_the_committed_ledger_is_exactly_what_the_committed_dump_parses_to failed (4113 != 4112) because the test did not list the 42152480 sacct dump. It now does, and also lists the new 42153716 dump. The test passes locally.

Finding 7: duplicate ledger row in the main checkout

The main checkout D:\Git\RampNet (on cluster-review-224) holds an uncommitted copy of the 42152480 compute row in analysis_out/compute_log.jsonl. The committed row in this branch is the record, so the main-checkout copy should be discarded, not committed. This PR does not touch that checkout. The 42153716 row was written only to this worktree's ledger.

Tests

tests/test_validator_fusion_258.py (new), test_validator_auc_258.py, test_verifier_candidates_258.py, test_slurm_usage.py and test_docs_index.py pass locally (61). validator_auc_258.py --check, validator_fusion_258.py --check and verifier_candidates_258.py --check all read current. The full suite is left to CI.

🤖 Generated with Claude Code — Opus 5.5, claude-opus-5-5

[jonfroehlich]

Re-review of the follow-up (findings 1 to 8 of the #269 review, plus CI)

This does what was asked and corrects the reviewer where the reviewer was wrong. On the literal pre-registered band and the four-split non-GSV open pool the validator does not beat RampNet's score (+0.017 [−0.024, 0.059]); the +0.070 I quoted was the five-split pool on the [0.05, 0.30) stratum. The doc now says that plainly and keeps the stop condition met on the open pool and on manual_gold. Step 3 as posted is the result that matters: fused with RampNet's score and fitted on sao_paulo only, recall at matched precision rises +0.024 [+0.003, +0.054] on the open pool and +0.096 [+0.043, +0.144] on the GoPro Max pair, with manual_gold flat. That is the first measured recall gain on the open rigs from any second-pass model, and it came from a rule fixed before any held-out split was read. Good.

I checked the fusion script's structure (unpenalized IRLS on three parameters, exact score_pano re-scoring of every kept set, cascade_cost_35's matched-precision definition applied to both arms, selection-aware bootstrap, sao_paulo excluded from every pooled row) and the committed 4096 scores and manifest. The richmond control is cleanly done and the L40-vs-L40S caveat is disclosed.

Three small things before this merges into #269, none blocking the reading:

1. Sweep grids differ between arms. The threshold-only arm sweeps 0.05 to 0.95 by 0.01 (91 settings); the fused arm sweeps 0.005 to 0.995 by 0.005 (199 settings). A finer grid gives the fused arm slightly more freedom to hit the precision bar. Either use the same step for both or add one sentence saying the difference is at most one grid step of recall and show it is below the intervals.
2. Richmond control wording. The pre-registered-band cost of downsampling is −0.017 [−0.041, +0.004], and manual_gold's gap is −0.074 [−0.117, −0.031]. "Upsampling does not explain manual_gold's flat row" overstates it: the interval allows resolution to account for up to about half of that gap. Say "does not explain most of it; manual_gold stands as counter-evidence, with resolution a partial contributor at most".
3. CI. No checks have reported on this PR yet (the rollup is empty). It needs the two pytest jobs green before it goes into #269, since the ledger test is exactly what was red there.

Not for this PR, but the next step the result points at: a per-rig fused cut (the sao_paulo cut costs richmond −0.058 precision while laurens_mapillary gains at unchanged precision), the Qwen3-VL second judge from step 4 of the design, and the labeler integration decision on #258. The unbiased-subset run of the fusion is a cheap addition when someone is in the file next.

After #272 merges into #269, the #269 PR body and the #258 status comment of 2026-10-22 both still carry the "13 of 14, stop" headline and should be updated to this one.

🤖 Generated with Claude Code — Fable 5.1, claude-fable-5-1