Repository navigation
aggregator: backtest forecasting models and report calibration #3986
Description
Parent chain: #1184 → #1256 → this issue. GitHub currently caps #1184 at 100 direct sub-issues and rejects additional links (REST 422), so implementation slices are nested under #1256, already a native child of #1184; no existing parent links were changed.
Design context: #1256; doc/2026-11-01T21-07-00Z_design_pm-stack_forecasting/2026-11-01T21-07-00Z_design_pm-stack_forecasting.md (§5). Depends on item history #3981 and forecast derivations #3982; include quota forecasts from #3983 when their outcome history is available.
Goal
Measure whether published ETA ranges and quota-wall forecasts are useful and calibrated, using only data available at each original forecast cutoff.
Acceptance criteria
- Store weekly as-of forecast snapshots with model version, cutoff, source watermarks, output intervals, confidence, and the minimal inputs needed to reproduce evaluation.
- Implement rolling-origin evaluation with no future leakage and compare against a documented trailing-rate/no-change baseline.
- Report sample counts, median absolute ETA error, P80 interval coverage, and quota-wall-before-reset classification accuracy by 2/4/8-week horizon and confidence grade.
- Do not label the model calibrated before at least 30 resolved forecasts across 8 weekly cutoffs; flag nominal P80 coverage outside 70–90% in two successive eligible windows.
- Tests prove cutoff isolation, model-version separation, unresolved outcome handling, and evaluation metrics on deterministic fixtures; document the disable/revert-to-unavailable rule for a persistently worse model.
Metadata
Metadata
Assignees
Labels
pm-stack (Pad/Grist) in developing-today/codeMedium: expected under 1 day of workNew feature or capability requestOrdinary priority (the default)Queue task is parked (waiting on CI, time, or an event)Filed or worked by the queue/worker pilotDeliberately paused with a recorded resume conditionQueue packet kind: verify, reproduce, and route this itemQueue worker lifecycle state
Activity
developing-today commentedon Nov 3, 2026
agent-note:v1
state: triaged (coverage pass)
Coverage pass 2026-11-03: reviewed in the fleet-wide per-issue pass (drift / overlap / already-handled / stale / blockers / ordering). Verdict: still live; queued in repo-health-code-pilot as triage at priority 40 (priority/normal). If this is already handled at current main, the triage closes it with evidence; if it needs splitting, the triage files the split.
Found via: coverage pass 2026-11-03
dezren39 commentedon Nov 3, 2026
agent-note:v1
state: parked (triaged by task 8339, lane code-n, direct lane)
task: 8339 · issue: #3986 · checked at main bb4624b, 2026-11-03
Triage findings:
Queue: task 8339 parked with ready condition "re-triage #3986 when #3981 and #3982 are closed". No fix is scoped yet because the inputs do not exist; the implementation slice should be cut from #1256 once #3982 lands.