Skip to content

aggregator: SnapshotStore accepts records it can never read back — non-finite estimate pairs and non-ISO timestamps pass validation, append reports success, and dedupe never sees them #1540

Closed

Description

@dezren39

aggregator: SnapshotStore accepts records it can never read back — non-finite estimate pairs and non-ISO timestamps pass validation, append reports success, and dedupe never sees them

Summary

isSnapshotRecord (pm-stack/aggregator/src/snapshots.ts:26-42) is the shared
structural validator for the usage snapshot store: the store's own
readAll/append, the feed ingest in sources/live.ts (which appends
feed.snapshots into the local store), and the writer scripts
(collect-fleet-feed.ts, gather-quota.ts, append-usage-snapshots.ts) all
gate on it. Two classes of record pass that validator but can never be read
back by the store, so SnapshotStore.append reports them as written, they
never appear in history, and — because dedupe is computed from readAll —
re-appending the same record appends it again every time.

Evidence (verified on main 3b55904, Bun 1.4.2, verbatim snapshots.ts sha256 49b14289 on the laptop)

Probe: probe-2026-10-27-laneB-snapshots (real snapshots.ts, only the
type-only model.ts import stubbed).

  1. Estimate pairs skip the finiteness check scalars get. Scalar
    value: NaN → isSnapshotRecord false (line 34: Number.isFinite), but
    value: { low: NaN, high: 10 } → true, and { low: Infinity, high: 10 }
    → true (lines 35-38 check only typeof === "number"). An inverted
    pair { low: 50, high: 10 } also passes.
    • append([NaN-pair]) returns 1; the stored line has
      "value":{"low":null,"high":10} (JSON.stringify turns NaN into null),
      which readAll then rejects — readAll() returns 0 records.
    • append([same NaN-pair]) again returns 1 (not 0): dedupe is built
      from readAll (snapshots.ts:99), which never sees the stored line, so
      every re-run of a writer appends another dead line and reports it as a
      new observation. The monthly file grows with records no reader will
      ever return.
  2. Any Date.parse-able ts passes, but file routing assumes an ISO
    prefix.
    snapshotFileFor (snapshots.ts:20-22) routes by
    ts.slice(0, 7), and readAll only reads files matching
    usage-(\d{4}-\d{2})\.jsonl (line 17). A record with
    ts: "October 20, 2026" passes isSnapshotRecord (line 29 only requires
    Date.parse to succeed), append returns 1 and writes it to
    usage-October.jsonl — a file readAll filters out, so readAll()
    returns 0. Same silent-loss + no-dedupe shape as (1).
  3. Control: a normal ISO record with a finite pair appends once, the
    second append returns 0 (dedupe works), and readAll returns 1.

Why it matters

The snapshot store is the append-only usage history the hub's history views
are built from, and its contract (module header, snapshots.ts:1-10) is that
writers and the backfill are "safe to re-run" because dedupe makes appends
idempotent. Records in either class above break that contract silently:
append counts them as stored history, no surface ever shows them, and every
writer re-run adds another copy. The pair case is reachable from any writer
whose estimate arithmetic can produce NaN/Infinity (e.g. a division by a
zero denominator upstream) — the validator is exactly the boundary that is
supposed to keep such values out, and it already does so for scalar values.

Acceptance criteria

  • isSnapshotRecord rejects estimate pairs whose low or high is not a
    finite number (mirroring the scalar branch), and rejects a pair with
    low > high.
  • isSnapshotRecord rejects a ts whose first 7 characters are not a
    YYYY-MM month the store can route (or, equivalently, snapshotFileFor
    output is guaranteed to match the readAll file pattern for every
    accepted record).
  • Regression tests: appending a NaN/Infinity-pair record and a non-ISO-ts
    record each append 0 and write nothing; the existing finite-pair control
    still appends once and dedupes on re-append.
  • No change to the on-disk format or to the dedupe key for valid records.

Notes

Activity

  1. developing-today commented on Oct 28, 2026

    @developing-today
    Contributor
  2. added
    kind/bugSomething isn't working (prefixed form of the classic bug label)
    queue/queuedHas a queued task in the Supabase agent queue
    source/creatorFiled by the new-issue-creator job
    task/triageQueue packet kind: verify, reproduce, and route this item
    priority/normalOrdinary priority (the default)
    app/aggregatorPackage/app: aggregator (pm-stack in this repo)
    effort/sSmall: expected under 2 hours of work
    on Oct 28, 2026
  3. dezren39 commented on Oct 28, 2026

    @dezren39
    MemberAuthor

    🤖 Agent note · filed

    • Agent: new-issue-creator job (new-issue-creator-code-b)
    • Session / ID: new-issue-creator-code-b run 2026-10-27 19:38 CDT (2026-10-28T00:47Z)
    • Found via: filed this run
    • Task: Triage the SnapshotStore validator/store contract mismatch: estimate pairs skip the finiteness check scalars get, and any Date.parse-able ts passes while file routing assumes an ISO month prefix — append reports success for records readAll can never return, and dedupe never sees them.
    • Expectation: effort/s — done when isSnapshotRecord rejects non-finite/inverted pairs and unroutable timestamps, with regression tests that appends of such records write nothing; valid finite-pair control unchanged.
    • Links: evidence probe (verbatim snapshots.ts sha256 49b14289, Bun 1.4.2 on the laptop) in the goal hidden_files probe-2026-10-27-laneB-snapshots/ (probe.ts + out.log); queue packet packet-1540.json (repo-health-code-pilot, triage)
  4. added
    queue/claimedQueue worker lifecycle state
    work/assignedQueue worker lifecycle state
    and removed
    queue/queuedHas a queued task in the Supabase agent queue
    on Oct 28, 2026
  5. 3 remaining items

  6. added
    queue/queuedHas a queued task in the Supabase agent queue
    work/assignedQueue worker lifecycle state
    queue/claimedQueue worker lifecycle state
    and removed
    queue/claimedQueue worker lifecycle state
    work/assignedQueue worker lifecycle state
    queue/queuedHas a queued task in the Supabase agent queue
    on Oct 28, 2026
  7. dezren39 commented on Oct 31, 2026

    @dezren39
    MemberAuthor

    🤖 Agent note · picked up

  8. added a commit that references this issue on Oct 31, 2026
  9. dezren39 commented on Oct 31, 2026

    @dezren39
    MemberAuthor

    🤖 Agent note · done — fixed & merged

    • Agent: queue worker (repo-worker-code), openchamber scheduled lane
    • Session / ID: queue task 1812 (small_fix)
    • Fix: PR fix(aggregator): reject snapshot records readAll can never return (#1540) #2925 merged as 96dd83a — isSnapshotRecord rejects non-finite/inverted pairs and non-routable ts; format + dedupe key unchanged
    • Tests: snapshot-writers.test.ts 14/14 pass (4 new); full suite 528 pass, 7 failures identical to clean tree (pre-existing pages area, unrelated)
    • Labels: removed work/assigned, queue/claimed, status/in-progress (terminal state)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

app/aggregatorPackage/app: aggregator (pm-stack in this repo)effort/sSmall: expected under 2 hours of workkind/bugSomething isn't working (prefixed form of the classic bug label)priority/normalOrdinary priority (the default)source/creatorFiled by the new-issue-creator jobtask/triageQueue packet kind: verify, reproduce, and route this item

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions