Repository navigation
[rush reporter][R8] Qualify AI reporter auto-selection #5981
Description
Parent: #5858
Goal
Prove that the AI reporter improves agent outcomes before enabling environment-based automatic selection.
Scope
- Build a deterministic corpus of representative install, configuration, dependency, build, cache, network/auth, and plugin failures.
- Compare legacy/plaintext/AI output for remediation success, context consumption, determinism, secret handling, and full-log discoverability.
- Validate built-in
COPILOT_CLIdetection plus configuredagentEnvironmentVariables. - Define pass/fail thresholds and store machine-readable gate results.
- Keep auto-selection disabled until the corpus gates pass.
Acceptance criteria
- The corpus is deterministic and runnable in CI.
- AI output never includes classified secrets and does not regress producer privacy.
- Thresholds for remediation quality, output size, determinism, and log-path correctness are documented and enforced.
- Passing the gate enables agent auto-selection only on the opt-in reporter path before Rush 6.
-
RUSH_REPORTER=legacyremains authoritative.
DAG
Unblocks R9.
Part of #5858.
Activity
TheLarkInn commentedon Aug 28, 2026
R8 qualification integration is reviewed and fixed in #5999 at ac3fd75, but this issue remains blocked.
The corpus now measures actual reporter output and AI log discoverability, validates exact context values/privacy markers, uses scenario-specific deterministic evidence with aggregate/per-case/compact size gates, preserves honest secret-diagnostic counts, and explicitly requires the separate privacy prerequisite before future AI eligibility. Three standalone runs were byte-identical; all blocking corpus gates passed locally, and reporter/rush-lib/apps-rush validation passed.
Remaining gates: #5990 must merge and be accepted; stacked base #5997 must merge; #5999 must receive required CI/review and merge with auto-merge still disabled while stacked; the qualification must pass on
main; only then may approved opt-in activation consume both prerequisites.RUSH_REPORTER=legacyremains authoritative and no Rush 6/default activation was introduced.