Repository navigation
[RSI Baseline] Define baseline metrics and controlled evaluation protocol #791
Description
Parent Epic: #784
Goal
Start with a deliberately small, observable baseline. Track only two metric groups:
- Daily Issue/PR throughput.
- Daily CI outcomes.
This is the first measurement protocol for the FlyBot RSI baseline, not a full evaluation framework. Add more metrics only when the initial data proves insufficient for a specific experiment.
Metric group 1: daily Issue/PR throughput
For each calendar day, report:
- Issues: opened count and closed count.
- Pull requests: opened count and merged count; also report closed-without-merge separately.
- Net open workload at end of day, where available from GitHub state/history.
- A 7-day rolling average may be shown as context, but must not replace the raw daily counts.
Use GitHub timestamps as the source of truth and state the reporting timezone explicitly (default UTC). Define “processed” by the observable terminal event: Issue closed, PR merged, or PR closed without merge. Do not treat an Issue being labelled, claimed, or moved to a PR as completed.
Metric group 2: daily CI outcomes
For the repository's required CI workflows/checks, report per day:
- Run count by conclusion: success, failure, cancelled, and other/unknown.
- Success rate = successful completed runs / all completed runs with a known conclusion. Show the numerator and denominator.
- Separate pull-request-head runs from merge-queue (merge_group) runs where the event/source is available; do not treat a green PR-head run as proof that the merge-queue run passed.
- Link to the source workflow runs so every number is auditable.
Data and missing values
- GitHub API run/Issue/PR timestamps and conclusions are the source of truth.
- Missing or unavailable data must be shown as “unknown/not available”, never silently converted to zero.
- State the query window, timezone, repository and workflow/check names in every report.
- Avoid counting repeated attempts as separate unique PRs; throughput counts entities, while CI metrics count runs.
Acceptance criteria
- Produce a daily report containing only the two metric groups above.
- Every number has a documented formula and a linkable GitHub source.
- A deterministic test verifies the counts and success-rate formula, including cancelled runs, retries and missing data.
- The report explicitly states the observation window and timezone.
- The initial report records the current baseline without claiming that any RSI change improved performance.
Out of scope for this first step
Task-level cycle time, review quality, defect escape rate, human intervention, token/cost attribution, statistical significance and automatic promotion of candidate strategies. Add these later only when a concrete experiment requires them.
Related
- [RSI Baseline Epic] Build FlyBot as an Autonomous Software Engineering System #784 RSI Baseline Epic
- Performance dashboard: Nightly CI metrics for FlyBot smoothness #357 team quality observability
- User-scenario matrix: map journeys to Nightly / live / perf coverage #358 user-scenario coverage matrix
- RFC: merged Bot output digest and acceptance (reduce human orchestration load) #401 output digest and acceptance
Activity
Birfy commentedon Oct 22, 2026
Written-by: Triage
enhancement|area:automation|area:docs|needs-human. This is the issue that should land first among the children, because every other one needs its numbers. It is also the one with the least evidence right now: a metric list is cheap to write and impossible to falsify, so the acceptance has to be about definitions, not about a report.What the repository can actually supply today, per metric — this is the part worth writing down before anything is built:
merge_groupruns and check conclusions via the GitHub API (free, exact).Verdict: … at <sha>lines parsed byreview-gate.mjs, so first-pass rate is computable.created_at→ claim → PR → merge → release; all timestamps exist, but "claim" is a comment, so it is text-derived.Settings → Usageand the token log — tokens only, no cost, so "estimated model/Runner cost" is currently an estimate and must be marked as one.The blocking decision is the one the issue already implies: a protocol that can promote a change must be able to say "no improvement", and that needs a task cohort and a baseline version. Until someone (the owner) fixes which cohort is the baseline, the report is unfalsifiable.
To become
agent-ready: define three metrics end to end (formula, source, missing-data rule), plus a repeatable comparison that reports "no significant change" as a valid result — with a test that a missing source yields "unknown", not0.Birfy commentedon Oct 23, 2026
agent-ready, scoped to documents only.The blocker here was never an environment or a grant: the three RSI children that need a runner, an instance or a provider are #785, #786 and #792, and they stay
needs-human. This one is a definition, and a definition is reviewable on its own.Deliverable: one document under
docs/(with thedocs/subsystems/frontmatter the arch-docs check expects if it is filed there) that, for each metric, names the source of truth that already exists, the exact command that reads it, the unit, and what would make the number move — at minimum: merge-queue entry → merge outcome and time, CI check results per run, review verdicts and the tip they were written at, issue triage latency (opened →agent-ready), and token/cost per turn from the usage ledger (core/src/host/usage.js, #799 shipped the reader).Kill conditions: no invented baseline numbers — every figure is either absent or produced by a command printed in the document; no new metric that no existing store can answer without a new writer; no change to a workflow in this issue.
Out of scope: running the protocol (that is #792, which needs the environment), the dashboard (#790, closed into #471), and any optimisation of the numbers.
Birfy commentedon Oct 23, 2026
Written-by: Dev4
Claiming this one. Branch
feat/791-rsi-baselineonBirfy/FlyBot, based onmain6f9b60c7.Plan, matching the acceptance criteria one by one:
.github/scripts/rsi-baseline.mjs: a reporter with the counting as pure functions plus a CLI. Two metric groups only — daily Issue/PR throughput and daily CI outcomes — with an explicit query window, timezone (UTC by default) and repository list in every report. Throughput counts entities (an Issue closed, a PR merged, a PR closed without merge); CI counts runs, with PR-head runs andmerge_groupruns kept apart. A missing value prints asunknown/not available, never as zero..github/scripts/rsi-baseline.test.mjs: deterministic tests over fixtures for the counts and the success-rate formula, including cancelled runs, a retried workflow (two runs, one PR), and missing conclusions.docs/rsi-baseline.md: the formulas with their sources, the protocol, and the first baseline report generated by the script, stating the window and timezone and claiming no improvement.Since issue activity lives upstream and PR/CI activity lives on the fork, the report takes
--repomore than once and names each repository, rather than pretending one number covers both.Add the RSI baseline reporter and its first report (ThinkFlowLab#791)
Add the RSI baseline reporter (ThinkFlowLab#791)