Skip to content

fix: correct 6 competitive-matrix drift-automation bugs (public-docs safety)#331

Merged
jpr5 merged 1 commit into
mainfrom
fix/competitive-matrix-bugs
Jul 23, 2026
Merged

fix: correct 6 competitive-matrix drift-automation bugs (public-docs safety)#331
jpr5 merged 1 commit into
mainfrom
fix/competitive-matrix-bugs

Conversation

@jpr5

@jpr5 jpr5 commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Why

scripts/update-competitive-matrix.ts runs weekly (.github/workflows/update-competitive-matrix.yml) and auto-commits to public, customer-facing docs (docs/index.html and docs/migrate-from-*/index.html). Six pre-existing bugs (all predate #328) caused it to write wrong data or silently skip updates. This PR fixes all six and adds real red→green regression coverage.

#328's single new FEATURE_RULES entry ("OpenRouter router / fallback simulation") and its migration-variant entry are left untouched and are covered by an explicit regression guard.

The 6 bugs & fixes

# Bug Fix
1 COMPETITOR_MIGRATION_PAGES mapped to docs/migrate-from-*.html, but the pages are directories (.../index.html), so existsSync always failed → every migration-page update silently skipped. mokksy/ai-mocks had no mapping at all. Paths → /index.html; added the mokksy/ai-mocks mapping. (Note: VidaiMock was already mapped; the genuinely missing competitor was mokksy/ai-mocks.)
2 Unanchored keyword regexes: cli→"client"/"click", sse→"assess" → false ✗→✓ flips in public docs. New keywordRegex() wraps each keyword in non-alphanumeric boundary lookarounds (?<![a-z0-9])…(?![a-z0-9]); works for literal tokens, path/regex keywords, and edge-punctuation keywords alike.
3 countProviders over-counted via substrings (cohere→"coherent", aws→"flaws") plus a redundant gemini.*interactions group double-counting Gemini. Bounded matching (same helper) + removed the redundant group (13 groups, each provider counted once).
4 String.replace(pattern, htmlString) let literal $, $&, $1 in HTML be interpreted → corrupted/duplicated output (3 sites). Switched the HTML-derived replacements to function form (…) => text so replacement text stays literal.
5 Migration feature-cell updater assumed the competitor column was adjacent to the label; the provider-count updater located it by header name — they disagreed when an intervening column (e.g. aimock) existed. Both now use one shared findMigrationCompetitorColumn() (token-aware header-name match — avoids the "vidaimock"⊇"aimock" substring trap).
6 Orphaned variant key "Realtime translate/whisper" never matched its rule label "Realtime transcription/translation" → that rule got no migration variants, cells never flipped. Renamed the key to match the rule label exactly.

Tests

A 1168-line competitive-matrix.test.ts already existed but reimplemented the script logic inline (a mirror), so it tested copies that mirrored the bugs — it could never catch them. Converted it to import the real script (added exports + an import.meta main-guard so importing doesn't run main()), fixed the one assertion that encoded the buggy behavior (countProviders all-groups 1413), and added red→green repros for each bug plus a regression guard for the OpenRouter entry.

Red → Green (fixture-based, no network)

RED — bug repros run against the pre-fix script (enabling refactor only, no behavioral change):

Tests  13 failed | 49 passed (62)

The 13 failures were exactly:

  • Bug 1: maps every mapped competitor to a file that actually exists on disk; includes a mapping for the mokksy/ai-mocks competitor
  • Bug 2: does not flip "CLI server" from the words "client"/"click"; does not flip "Chat Completions SSE" from the word "assess"
  • Bug 3: does not count "cohere" inside "coherent" or "aws" inside "flaws"; counts Gemini exactly once; counts all 13 provider groups …
  • Bug 4: applyChanges does not duplicate the row when replacement text contains $&; updateProviderCounts does not corrupt the table when a cell contains $&
  • Bug 5: flips the competitor cell even when aimock is the first data column; resolves the column when the header text differs from the competitor key (Mokksy)
  • Bug 6: returns variants for the real rule label 'Realtime transcription/translation'; flips a migration row that uses a variant label for the realtime rule

(The #328 OpenRouter guard tests and the positive-control tests passed in the RED run, as expected.)

GREEN — after the 6 fixes:

✓ src/__tests__/competitive-matrix.test.ts (62 tests)
Test Files  1 passed (1)
      Tests  62 passed (62)

Full suite / build / lint (no regressions):

Tests  4706 passed | 44 skipped (4750)     # vitest run
Build complete                              # pnpm build (tsdown)
All matched files use Prettier code style!  # pnpm format:check
eslint .  → 0 problems                      # pnpm lint

Live end-to-endnpx tsx scripts/update-competitive-matrix.ts --dry-run against real GitHub still fires main(), parses all 4 competitors + the 38-row / 6-column matrix, and reports "No changes detected" (no spurious flips).

Scope / safety

  • No package version bump (this is scripts/docs tooling, not the published package surface).
  • #328's FEATURE_RULES + variant entry unchanged; guarded by regression tests.

🤖 Generated with Claude Code

https://claude.ai/code/session_01KamK73Wu5heLxJEogM6hHC

@pkg-pr-new

pkg-pr-new Bot commented Jul 23, 2026

Copy link
Copy Markdown

Open in StackBlitz

npm i https://pkg.pr.new/@copilotkit/aimock@331

commit: 0d09c50

…safety)

The weekly update-competitive-matrix.ts auto-commits to public docs
(docs/index.html + docs/migrate-from-*/index.html). Six pre-existing bugs
wrote wrong data or silently skipped updates:

1. Migration-page paths pointed at docs/migrate-from-*.html but the pages are
   directories (index.html); every migration update was silently skipped.
   Fixed paths and added the missing mokksy/ai-mocks mapping.
2. Unanchored keyword regexes flipped cells on substrings ("cli"->"client",
   "sse"->"assess"). Added non-alphanumeric boundary lookarounds.
3. countProviders over-counted via substrings ("cohere"->"coherent",
   "aws"->"flaws") and a redundant gemini group. Bounded matching + deduped.
4. String.replace let literal dollar sequences in HTML be interpreted. Switched
   the HTML-derived replacements to function form so text stays literal.
5. Migration feature-cell updater assumed the competitor was the first data
   column while the count updater used header-name lookup. Both now share one
   header-name column resolver.
6. Orphaned variant key "Realtime translate/whisper" never matched its rule
   "Realtime transcription/translation"; renamed so the rule gets variants.

Tests: converted the pre-existing mirror test file to import the real script
(exports added + import.meta main-guard) and added red-green repros for each
bug plus a regression guard for the OpenRouter router/fallback entry.
@jpr5
jpr5 force-pushed the fix/competitive-matrix-bugs branch from 28e81b4 to 0d09c50 Compare July 23, 2026 01:56
@jpr5
jpr5 marked this pull request as ready for review July 23, 2026 01:59
@jpr5
jpr5 merged commit d089930 into main Jul 23, 2026
23 checks passed
@jpr5
jpr5 deleted the fix/competitive-matrix-bugs branch July 23, 2026 01:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant