fix: descope fal live-queue drift leg from CI base (unpoison drift baseline) - #330
Merged
Conversation
commit: |
jpr5
added a commit
that referenced
this pull request
Jul 23, 2026
… leg The live fal queue-lifecycle canary (#327) assumed jobs sit in IN_QUEUE long enough to cancel for $0. Funded flux/schnell returns status:"IN_PROGRESS" on submit immediately; by cancel time the job is running/completed and fal's cancel envelope is {detail:...} (no status field). The leg hard-asserted `typeof cancel.body.status === 'string'` -> AssertionError -> collector exit-5 quarantine -> the drift-live-pr base step red on every PR (descoped in #330). Fix (probe-only, no fal.ts change): - Lifecycle: state-agnostic submit -> status -> CANCEL; accept any valid queued/running/completed status; cancel always fires even if status polls throw. Relax the brittle cancel assertion to require only a JSON envelope ({status} OR {detail}); leave shape grading to triangulate so a genuine 2xx-wrong-shape still reports critical drift. - Infra-skip (folds in #329, which is being closed): 401/402/403 locked-or- exhausted/429/5xx raise a distinguished FalCanarySkip -> honest ctx.skip, never an InfraError that quarantines the baseline. - Re-enable: drop the FAL_LIVE_QUEUE opt-in gate #330 added so the corrected leg runs live in CI again (still gated on FAL_KEY). Cost-safe: cheapest model, cancel ASAP, result endpoint never fetched. Residual: <= one sub-cent generation if the model races to completion. Red-green in fal-canary-skip.test.ts drives the real canary + real triangulate: IN_PROGRESS+{detail} handled (leg passes), 403 -> FalCanarySkip, 2xx-wrong- shape still critical.
jpr5
added a commit
that referenced
this pull request
Jul 23, 2026
… leg The live fal queue-lifecycle canary (#327) assumed jobs sit in IN_QUEUE long enough to cancel for $0. Funded flux/schnell returns status:"IN_PROGRESS" on submit immediately; by cancel time the job is running/completed and fal's cancel envelope is {detail:...} (no status field). The leg hard-asserted `typeof cancel.body.status === 'string'` -> AssertionError -> collector exit-5 quarantine -> the drift-live-pr base step red on every PR (descoped in #330). Fix (probe-only, no fal.ts change): - Lifecycle: state-agnostic submit -> status -> CANCEL; accept any valid queued/running/completed status; cancel always fires even if status polls throw. Relax the brittle cancel assertion to require only a JSON envelope ({status} OR {detail}); leave shape grading to triangulate so a genuine 2xx-wrong-shape still reports critical drift. - Infra-skip (folds in #329, which is being closed): 401/402/403 locked-or- exhausted/429/5xx raise a distinguished FalCanarySkip -> honest ctx.skip, never an InfraError that quarantines the baseline. - Re-enable: drop the FAL_LIVE_QUEUE opt-in gate #330 added so the corrected leg runs live in CI again (still gated on FAL_KEY). Cost-safe: cheapest model, cancel ASAP, result endpoint never fetched. Residual: <= one sub-cent generation if the model races to completion. Red-green in fal-canary-skip.test.ts drives the real canary + real triangulate: IN_PROGRESS+{detail} handled (leg passes), 403 -> FalCanarySkip, 2xx-wrong- shape still critical.
jpr5
added a commit
that referenced
this pull request
Jul 23, 2026
… leg (#332) ## What Proper fix for the fal live queue-lifecycle drift probe (`falQueueLifecycleCanary` + the live block in `src/__tests__/drift/fal-queue.drift.ts`, merged #327), and **re-enables** the live leg that #330 temporarily descoped. Probe-only — no change to aimock's `fal.ts` core. ## The bug (verified live in CI, root cause) The probe assumed fal jobs sit in `IN_QUEUE` long enough to cancel for \$0. Funded `flux/schnell` returns `status:"IN_PROGRESS"` on the submit envelope **immediately**. By cancel time the job is running/completed, so fal's cancel envelope is `{detail:"..."}` — **no `status` field**. The live leg hard-asserted `typeof cancel.body.status === "string"` → AssertionError → the collector could not map it → **exit-5 quarantine** → the `drift-live-pr` **base** step (main's code) went red on every PR. #330 was the fast, reversible descope; this is the proper fix. ## The fix 1. **Lifecycle** — state-agnostic submit → status → CANCEL. Accept any 2xx submit and any valid queued/running/completed status (`IN_QUEUE` **or** `IN_PROGRESS` …), never assert the job is specifically queued. Cancel still fires even if the status poll throws (a job is never left running). 2. **Envelope shapes / drift** — the cancel assertion now requires only a JSON envelope (`{status}` **or** `{detail}`); precise shape grading is left to `triangulate` against the mock, so a genuine **2xx-wrong-shape still reports critical drift**. 3. **Infra-skip (folds in #329, which is being closed in favor of this PR)** — `401/402/403` (locked/exhausted) `/429/5xx` raise a distinguished `FalCanarySkip` → honest `ctx.skip`, never an `InfraError` that quarantines the baseline. Includes #329's `fal-canary-skip.test.ts` guard tests. 4. **Re-enable** — removes the `FAL_LIVE_QUEUE` opt-in gate #330 added; the corrected leg runs live in CI again (still gated on `FAL_KEY`). ## Cost safety Cheapest reliably-available model (`fal-ai/flux/schnell`), cancel ASAP, the paid `response_url` result endpoint is **never** fetched. Residual: at most **one sub-cent** generation if the model races to completion before the immediate cancel lands. ## Red-green proof (local, drives the real canary + real `triangulate`) RED captured against pre-fix probe (origin/main @ d43701c): ``` × RED(a): IN_PROGRESS submit + {detail} cancel — old assert `typeof cancel.body.status === "string"` → AssertionError: expected 'undefined' to be 'string' × RED(b): 403 user-locked at submit → rejects with InfraError: "INFRA_ERROR: fal Queue Lifecycle …" (quarantine-causing, not a skip) ``` GREEN post-fix (`src/__tests__/drift/fal-canary-skip.test.ts`, 4/4 pass): - (a) IN_PROGRESS submit + `{detail}` cancel → canary handles it, `triangulate` reports no critical → leg passes. - (b) 403 user-locked → `FalCanarySkip` (not `InfraError`); 401/402 also skip. - (c) **guard**: 2xx submit whose envelope diverges from the mock → still critical drift (fix did not swallow genuine drift). ## Checks (local) `prettier` ✓ · `eslint` ✓ · `tsc --noEmit` (tests-inclusive) 0 errors ✓ · `test:drift` 16 passed / 9 skipped ✓ · full suite 4693 passed ✓ · `tsdown` build ✓ Live validation (corrected leg against funded fal) = CI `drift-live-pr` on this PR — see checks below. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01KamK73Wu5heLxJEogM6hHC
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Gates OFF the fal LIVE queue-lifecycle drift leg so it no longer runs in CI / the shared base drift collector, while KEEPING all of fal's static drift coverage.
The live probe (
falQueueLifecycleCanary, driven by the live block insrc/__tests__/drift/fal-queue.drift.ts, merged in #327) fails against real fal:flux/schnellreturnsstatus:"IN_PROGRESS"immediately (not the assumedIN_QUEUE) → AssertionError → collector exit-5 quarantine →drift-live-prbase step (which runs main's code) reds on EVERY PR, blocking #324 and #325 from merging.A proper fix for the probe is being built in parallel (
fix/fal-probe-in-progress). This PR is the fast, reversible descope so coverage PRs can land now.The change (one hunk)
Added an explicit opt-in env gate to the live
describe— it now skips unless BOTH are set:FAL_LIVE_QUEUEis not set in CI, so the live leg is skipped everywhere. Static fal tests (fal.drift.ts+ the mock-vs-exemplar tests infal-queue.drift.ts) are untouched. Re-enable by removing theFAL_LIVE_QUEUEgate once the probe is fixed.Red / green (local)
RED (unmodified main,
FAL_KEYset) — live leg runs and fails:GREEN (this PR,
FAL_KEYset, noFAL_LIVE_QUEUE) — live leg skips, no AssertionError:Re-enable path verified (
FAL_KEY+FAL_LIVE_QUEUEboth set) — live leg runs again.Full
pnpm run test:drift(CI-style, noFAL_KEY): 93 passed | 65 skipped. Typecheck, prettier, eslint, build all clean.Expected CI note (fix-forward)
This PR's own
drift-live-prwill be RED at the base step because base runs main's still-broken fal probe — that red is exactly the poison this PR removes, and it cannot go green pre-merge (base runs old main). All other checks are expected green. Merging fix-forward heals main so subsequent PRs' base steps no longer run the fal live leg.🤖 Generated with Claude Code