Skip to content

rsi-baseline.mjs stops silently at 2000 items and files skipped under other, so a day can be understated and its success rate overstated #814

Closed
Birfy/FlyBot
#38

Description

@Birfy

What is wrong

Two independent under-counts in .github/scripts/rsi-baseline.mjs.

  1. Silent cap at 2000 items. allPages (:220-229) loops for (let page = 1; page <= 20; page += 1) at per_page=100 and stops when a page comes back with fewer than 100 items. When page 20 is full the loop simply ends: 2000+ items are returned as if they were all of them, and neither the report nor the exit code says a truncation happened. Both collectRepo (:240) and collectRuns (:270) read through it, so a busy day can understate throughput and run counts.

  2. skipped is filed under other. summarizeRuns (:103-107) maps every conclusion outside success/failure/cancelled to other, and rates (:117-123) keeps other in the denominator. Reproduced against the module as it stands:

     summarizeRuns({ runs: [3 × success, 2 × skipped] })
     → all {"success":3,"failure":0,"cancelled":0,"other":2,"unknown":0}
       denominator 5, successRate 0.6
    

    A run that did nothing counts against the success rate. It is the distinction Live E2E has failed on every scheduled run since it landed: the repository secret DEEPSEEK_API_KEY is not configured #794 is about: a skipped run measured nothing and must not move the number in either direction.

Expected behaviour

  • A capped fetch is visible in the report as "at least 2000 / possibly truncated" on that repo, never as a clean total.
  • skipped has its own bucket in all and in every byEvent entry, its own report column, and is excluded from the denominator and from successRate.
  • docs/rsi-baseline.md states both rules.

Acceptance

  1. summarizeRuns returns skipped separately: a day of 3 success + 2 skipped gives skipped: 2, denominator: 3, successRate: 1.
  2. The denominator is success + failure + cancelled + other; skipped and unknown stay out of it, pinned by a test.
  3. renderReport prints a Skipped column, and its formula line names the exclusion.
  4. The fetch exposes truncated: true when page 20 came back full, and the report prints that marker for the repository; a fetch ending on a short page (e.g. page 20 with 99 items) does not set it.
  5. docs/rsi-baseline.md is updated for the column and the denominator.
  6. Tests in .github/scripts/rsi-baseline.test.mjs: a full 20-page sequence → truncated; 1999 items → not truncated; the skipped bucket; the denominator arithmetic.
  7. The report still runs end to end against the live API — node .github/scripts/rsi-baseline.mjs --repo ThinkFlowLab/FlyBot --day <a recent day> --out /tmp/rsi.md produces a file with the new column.

Kill conditions

  • Marking truncation only when exactly 2000 items arrive. A full page 20 means "2000 or more"; both cases must be marked.
  • Counting skipped in the denominator, or folding it back into other.
  • Printing a truncated number without the marker: a silently smaller number is the bug.
  • Changing what unknown or a cancelled run means; both are deliberate (:114-116).

Out of scope

Activity

  1. added
    bugSomething isn't working
    agent-readyMaintainer-approved spec: an agent may take it. Only maintainers set this; never in a template
    area:automationScheduled tasks and automation
    on Oct 23, 2026
  2. Birfy commented on Oct 23, 2026

    @Birfy
    CollaboratorAuthor

    Written-by: Dev4

    Claiming this one. Branch fix/814-rsi-pagination-buckets on Birfy/FlyBot, cut from f20cfa37f9f2ffd07fa1387eece36756fbd49dbb.

    Scope is exactly the two under-counts plus the three sentences of protocol they need: allPages reports a capped fetch instead of ending silently, and skipped gets its own bucket, column and denominator rule. Nothing about timed_out/action_required/stale (they stay in other) and nothing about the search endpoints.

    One thing this changes beyond the code: the checked-in report docs/reports/2026-10-22-rsi-baseline.md was produced by the old bucketing, so it will be regenerated with the new column rather than left to disagree with the protocol.

  3. added
    agent-workingAn agent has claimed this issue (the claim comment names the branch)
    and removed
    agent-readyMaintainer-approved spec: an agent may take it. Only maintainers set this; never in a template
    on Oct 23, 2026
  4. removed
    agent-workingAn agent has claimed this issue (the claim comment names the branch)
    on Oct 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:automationScheduled tasks and automationbugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions