Skip to content

aggregator(charges-match): dedupe collapses distinct charges — same-source repeats within 3 days merge into one row, and empty-normalized merchants merge across merchants #2130

New issue
New issue
Closed
#4098
Closed
aggregator(charges-match): dedupe collapses distinct charges — same-source repeats within 3 days merge into one row, and empty-normalized merchants merge across merchants#2130
#4098
Assignees
dezren39
Labels
app/aggregatorPackage/app: aggregator (pm-stack in this repo)app/pm-stackpm-stack (Pad/Grist) in developing-today/codeeffort/xsTrivial: expected under 30 minutes of workkind/bugSomething isn't working (prefixed form of the classic bug label)priority/normalOrdinary priority (the default)source/creatorFiled by the new-issue-creator jobsource/design-architectFiled by the repo design architect jobtask/triageQueue packet kind: verify, reproduce, and route this item

Description

@dezren39
dezren39
opened on Oct 29, 2026

Summary

matchCharges in pm-stack/aggregator/src/charges-match.ts (merged in PR #2113 for #1057) dedupes on (amountUsd, normalizedMerchant, ±3 days) only — it never requires the two candidates to come from different sources, and it never rejects an empty normalized merchant. Two distinct real charges therefore collapse into one row, under-counting the charges strip that #1057 exists to automate.

The module's own contract is narrower: the header says dedupe is across sources ("Deduped across sources", "Charges from two sources for the same purchase post within days", charges-match.ts:4-9, :54, :178-182), and #1057's acceptance is "a charge is never double-counted across sources".

Evidence (verified this run, main 58621026, laptop probe ~/git/fleet/probe-2026-10-29-laneB-charges/probe.ts importing the shipped file)

Control — cross-source pair dedupes correctly:

  • Plaid CURSOR.COM $20 2026-10-27 + Gmail Cursor $20 2026-10-28 → 1 row (expected).

Bug A — same-source distinct charges collapse:

  • Two Plaid candidates, CURSOR.COM $20 on 2026-10-27 and 2026-10-28 → 1 row (real charges: 2). Code path: charges-match.ts:190-202 finds the existing cluster by amount/normalized/window regardless of source; the second Plaid candidate sets neither existing.plaid (already set) nor existing.gmail, and is silently dropped (continue).
  • Two Gmail candidates, same merchant/amount/day → 1 row (real charges: 2).

This is not exotic: usage-based AI/API charges, per-seat charges, and two purchases from the same vendor within three days at the same price are exactly the charges this strip reports.

Bug B — empty-normalized merchants collide across merchants:

  • normalizeMerchantName("1234567") === "" and normalizeMerchantName("9876543") === "" (digit-only tokens are dropped, charges-match.ts:84-93). A Plaid charge from merchant 1234567 ($9.99) and a Gmail alert from the different merchant 9876543 ($9.99, same day) → 1 row labelled 1234567. Any two descriptors that normalize to "" with equal amount within the window merge, whatever they are.

Why it matters

The charges strip is financial evidence: under-counting hides real spend (the failure mode is silent — no error, just a missing row), and Bug B can attribute one vendor's charge to another. Rerun determinism (a stated design goal) makes the wrong answer stable, so it will not self-correct on the next collector run.

Acceptance criteria

  • Dedupe only merges candidates from different sources (a cluster holds at most one Plaid and one Gmail candidate); two same-source candidates with the same merchant/amount inside the window produce two rows.
  • Candidates whose normalized merchant is empty never merge with a different raw merchant; they are kept as their own rows (or explicitly handled and tested).
  • The existing cross-source control case still produces one Plaid-of-record row, and all existing charges-match.test.ts tests pass.
  • New tests in pm-stack/aggregator/test/charges-match.test.ts cover: same-source repeats inside the window, and two distinct empty-normalizing descriptors with equal amounts.

Related: #1057 (the feature this shipped under — distinct from its cross-source double-count acceptance), #1047 (epic).

Activity

  1. developing-today commented on Oct 29, 2026

    @developing-today
    developing-todayBot
    on Oct 29, 2026 – with developing.today
    Contributor

    CODE-1028

  2. dezren39
    added
    kind/bugSomething isn't working (prefixed form of the classic bug label)
    priority/normalOrdinary priority (the default)
    queue/queuedHas a queued task in the Supabase agent queue
    source/creatorFiled by the new-issue-creator job
    task/triageQueue packet kind: verify, reproduce, and route this item
    status/needs-triageNot yet triaged; needs a first look
    app/aggregatorPackage/app: aggregator (pm-stack in this repo)
    effort/xsTrivial: expected under 30 minutes of work
    on Oct 29, 2026

  3. dezren39 commented on Oct 29, 2026

    @dezren39
    dezren39
    on Oct 29, 2026
    MemberAuthor

    🤖 Agent note · filed

    • Agent: new-issue-creator (new-issue-creator-code-b)
    • Session / ID: cron new-issue-creator-code-b, 2026-10-29 00:50 CDT run
    • Found via: filed this run
    • Task: Filed this issue — charges-match dedupe collapses distinct same-source charges and empty-normalized merchants; verified by a laptop probe importing the shipped charges-match.ts on main 5862102.
    • Expectation: effort/xs triage — pin the cross-source-only dedupe contract, then route the small fix.
    • Links: packet queue-pilot/packet-2130-triage.json (repo-health-code-pilot)
  4. github-actions
    assigned
    dezren39
    on Oct 29, 2026
  5. dezren39
    added a parent issue on Oct 31, 2026
  6. dezren39
    added
    source/design-architectFiled by the repo design architect job
    on Oct 31, 2026

  7. dezren39 commented on Oct 31, 2026

    @dezren39
    dezren39
    on Oct 31, 2026
    MemberAuthor

    🤖 Design-architect pass 27 (2026-10-31) — agent note: this issue is adopted as part of Phase 1 (Identity) of design #3161 ("a charge is evidence"). Triage: do NOT re-file, re-scope or split it — record the phase structure instead. The design records the shared identity rule this phase lands under: same charge only when provenance differs (cross-source for candidate dedupe; hand-fed vs auto for the merge), amounts equal, dates within the existing 3-day window, merchant identity at token boundaries (exact normalized equality for dedupe; token-boundary containment for the hand-fed merge, preserving the documented METAPAY / "Meta PAYG spill" pair), an empty normalized merchant never matches, and conflicting known payment methods veto the merge. Phase 1 siblings: #2130, #3049, #3123. Details on #3161 and in files/designs/code-charges-pipeline-contract.md (goal workspace).

  8. dezren39
    added
    work/assignedQueue worker lifecycle state
    on Nov 1, 2026

  9. 26 remaining items

  10. dezren39
    added
    status/in-progressQueue worker lifecycle state
    and removed
    status/parkedDeliberately paused with a recorded resume condition
    queue/parkedQueue task is parked (waiting on CI, time, or an event)
    on Nov 2, 2026

  11. dezren39 commented on Nov 2, 2026

    @dezren39
    dezren39
    on Nov 2, 2026
    MemberAuthor

    🤖 Agent note · picked up

  12. dezren39
    added
    queue/parkedQueue task is parked (waiting on CI, time, or an event)
    status/parkedDeliberately paused with a recorded resume condition
    and removed
    status/in-progressQueue worker lifecycle state
    queue/claimedQueue worker lifecycle state
    on Nov 2, 2026

  13. dezren39 commented on Nov 2, 2026

    @dezren39
    dezren39
    on Nov 2, 2026
    MemberAuthor

    🤖 Agent note · parked

    • Agent: repo-worker-code (OpenChamber scheduled lane); task 5186.
    • PR fix(aggregator): dedupe charges only across sources (#2130) #4098 remains OPEN and Drew-authored; this lane will not touch or merge it. The pm-aggregator tests check is still QUEUED; CodeRabbit is neutral/skipped.
    • Parked for 20 minutes and ending this launch per the CI rule. Resume with fresh PR/check state; keep the Drew-owned PR untouched. Prior targeted regression tests passed 13/13.
  14. dezren39
    added a commit that references this issue on Nov 2, 2026

    fix(aggregator): dedupe charges only across sources (#2130) (#4098)

    bc0afe2
  15. dezren39
    closed this as completedin #4098on Nov 2, 2026

  16. dezren39 commented on Nov 2, 2026

    @dezren39
    dezren39
    on Nov 2, 2026
    MemberAuthor

    Done by queue worker lane D (task 5186): PR #4098 merged to main as bc0afe2; issue #2130 auto-closed by the merge.

    Verification at merge time (fresh reads this shift):

    • PR fix(aggregator): dedupe charges only across sources (#2130) #4098 diff = exactly the two in-scope files (pm-stack/aggregator/src/charges-match.ts, pm-stack/aggregator/test/charges-match.test.ts), +29/-0, MERGEABLE.
    • CI run 41369251101 on head 62cbe9e: all charges-match tests pass (including the new same-source Plaid/Gmail and empty-merchant regression tests); suite total 950 tests, 948 pass.
    • The only 2 failures — "slice-3 routes > dns + sessions + pages routes serve" and "hub renders > service detail: full metrics, chart, reads, conflicts" — are the pre-existing unrelated failures the previous claim verified as reproduced on clean main; they are outside this issue's scope.
    • No Codex review findings exist on PR fix(aggregator): dedupe charges only across sources (#2130) #4098 (no reviews, no inline comments), so the Codex gate had nothing undispositioned.
    • Note on ownership: the packet carries no Drew-owned/held flag (risk flags empty) and PR fix(aggregator): dedupe charges only across sources (#2130) #4098 is this task's canonical PR on branch queue/5186-2130-dedupe-charges, so it was merged under the standing merge directive once verified — the earlier "Drew-authored" park classification came from the author login alone, which every fleet PR shares.
  17. dezren39
    removed
    status/parkedDeliberately paused with a recorded resume condition
    queue/parkedQueue task is parked (waiting on CI, time, or an event)
    work/assignedQueue worker lifecycle state
    on Nov 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

app/aggregatorPackage/app: aggregator (pm-stack in this repo)app/pm-stackpm-stack (Pad/Grist) in developing-today/codeeffort/xsTrivial: expected under 30 minutes of workkind/bugSomething isn't working (prefixed form of the classic bug label)priority/normalOrdinary priority (the default)source/creatorFiled by the new-issue-creator jobsource/design-architectFiled by the repo design architect jobtask/triageQueue packet kind: verify, reproduce, and route this item

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions