Skip to content

feat: fal.ai live queue-lifecycle drift probe (cost-safe, FAL_KEY-gated) - #327

Merged
jpr5 merged 1 commit into
mainfrom
feat/fal-live-queue-drift
Jul 23, 2026
Merged

feat: fal.ai live queue-lifecycle drift probe (cost-safe, FAL_KEY-gated)#327
jpr5 merged 1 commit into
mainfrom
feat/fal-live-queue-drift

Conversation

@jpr5

@jpr5 jpr5 commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

What

Adds a live queue-lifecycle drift probe for the fal.ai queue surface (src/fal.ts). Rows 25/26 of the drift-coverage audit were STATIC-only; this closes the live gap for the queue surface with a cost-safe canary that self-activates once FAL_KEY is mirrored to repo secrets.

Mirrors the OpenRouter-video precedent (#323): live leg is describe.skipIf(!FAL_KEY), so it skips in CI until the secret lands, then runs automatically.

Changes

  • src/__tests__/drift/providers.ts — new falQueueLifecycleCanary(apiKey, modelId, input): drives the REAL fal queue API through submit → status → immediate cancel, returning the three envelope bodies. Cancel is issued even if the status poll throws (never leaves a job to RUN).
  • src/__tests__/drift/fal-queue.drift.ts — new describe.skipIf(!FAL_KEY)("fal.ai queue lifecycle (live, cost-safe)") block: asserts the load-bearing submit/status/cancel fields on the real envelope and triangulates each step (exemplar × real × aimock mock) under the fal-queue surface slug. Completed-result envelope stays STATIC-only (never fetched — that is the only paid retrieval).
  • .github/workflows/test-drift.yml + fix-drift.yml — add FAL_KEY: ${{ secrets.FAL_KEY }} to every drift-job env block, alongside the existing provider keys (minimal/additive).

Surface registry already contains the fal-queue slug — no registry edit needed.

Cost note

fal bills compute only when a queued job RUNS. The canary submits (free) and cancels immediately while the job is still IN_QUEUE, so expected cost is $0. It never fetches the result endpoint. Cheapest reliably-available model (fal-ai/flux/schnell) bounds the worst case; residual exposure is at most one sub-cent generation in the rare race where the model completes before the cancel lands.

Red-green

Live red-green cannot run locally (FAL_KEY is not yet a repo secret and 1Password access is guard-blocked) — CI will capture the live red-green once FAL_KEY is mirrored. Structural red-green proven locally against the real drift machinery:

RED — forced the live block to run with a deterministically drifted real submit envelope (queue_position as string vs the mock's number):

API DRIFT DETECTED: fal.ai queue submit (live)
  Surface: fal-queue

  1. [critical] TYPE MISMATCH between real API and mock: string vs number
     Path:    queue_position
     SDK:     number
     Real:    string
     Mock:    number
Test Files  1 failed (1)

GREEN — reverted; live block skips (no FAL_KEY), static legs pass:

✓ src/__tests__/drift/fal-queue.drift.ts (5 tests | 1 skipped)
Test Files  1 passed (1)

Full suite: Test Files 16 passed | 9 skipped (25) Tests 93 passed | 65 skipped. Typecheck (touched files), prettier, eslint, build all clean.

🤖 Generated with Claude Code

@pkg-pr-new

pkg-pr-new Bot commented Jul 22, 2026 •
edited
Loading

Copy link
Copy Markdown

Open in StackBlitz

npm i https://pkg.pr.new/@copilotkit/aimock@327

commit: ab42ab5

@jpr5
jpr5 marked this pull request as ready for review July 23, 2026 00:09
@jpr5
jpr5 force-pushed the feat/fal-live-queue-drift branch from 9932fda to ab42ab5 Compare July 23, 2026 00:09
@jpr5
jpr5 merged commit 3156209 into main Jul 23, 2026
29 checks passed
@jpr5
jpr5 deleted the feat/fal-live-queue-drift branch July 23, 2026 00:11
This was referenced Jul 23, 2026
Closed
Merged
jpr5 added a commit that referenced this pull request Jul 23, 2026
@jpr5
fix: descope fal live-queue drift leg from CI base (unpoison drift ba…
d43701c 
…seline) (#330)

## What

Gates OFF the fal LIVE queue-lifecycle drift leg so it no longer runs in
CI / the shared base drift collector, while KEEPING all of fal's static
drift coverage.

The live probe (`falQueueLifecycleCanary`, driven by the live block in
`src/__tests__/drift/fal-queue.drift.ts`, merged in #327) fails against
real fal: `flux/schnell` returns `status:"IN_PROGRESS"` immediately (not
the assumed `IN_QUEUE`) → AssertionError → collector exit-5 quarantine →
`drift-live-pr` **base** step (which runs main's code) reds on EVERY PR,
blocking #324 and #325 from merging.

A proper fix for the probe is being built in parallel
(`fix/fal-probe-in-progress`). This PR is the fast, reversible descope
so coverage PRs can land now.

## The change (one hunk)

Added an explicit opt-in env gate to the live `describe` — it now skips
unless BOTH are set:

```ts
describe.skipIf(!FAL_KEY || !process.env.FAL_LIVE_QUEUE)("fal.ai queue lifecycle (live, cost-safe)", ...)
```

`FAL_LIVE_QUEUE` is not set in CI, so the live leg is skipped
everywhere. Static fal tests (`fal.drift.ts` + the mock-vs-exemplar
tests in `fal-queue.drift.ts`) are untouched. Re-enable by removing the
`FAL_LIVE_QUEUE` gate once the probe is fixed.

## Red / green (local)

**RED** (unmodified main, `FAL_KEY` set) — live leg runs and fails:
```
× fal.ai queue lifecycle (live, cost-safe) > real submit + status + cancel ...
  → INFRA_ERROR: fal queue submit: API returned 401 ...
Tests  1 failed | 4 passed (5)
```

**GREEN** (this PR, `FAL_KEY` set, no `FAL_LIVE_QUEUE`) — live leg
skips, no AssertionError:
```
✓ src/__tests__/drift/fal-queue.drift.ts (5 tests | 1 skipped) 22ms
Tests  4 passed | 1 skipped (5)
```

**Re-enable path verified** (`FAL_KEY` + `FAL_LIVE_QUEUE` both set) —
live leg runs again.

Full `pnpm run test:drift` (CI-style, no `FAL_KEY`): **93 passed | 65
skipped**. Typecheck, prettier, eslint, build all clean.

## Expected CI note (fix-forward)

This PR's own `drift-live-pr` will be RED at the **base** step because
base runs main's still-broken fal probe — that red is exactly the poison
this PR removes, and it cannot go green pre-merge (base runs old main).
All other checks are expected green. Merging fix-forward heals main so
subsequent PRs' base steps no longer run the fal live leg.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
@jpr5 jpr5 mentioned this pull request Jul 23, 2026
Merged
jpr5 added a commit that referenced this pull request Jul 23, 2026
@jpr5
fix: fal probe handles IN_PROGRESS + infra-skip; re-enable live queue…
ce57d9e 
… leg

The live fal queue-lifecycle canary (#327) assumed jobs sit in IN_QUEUE long
enough to cancel for $0. Funded flux/schnell returns status:"IN_PROGRESS"
on submit immediately; by cancel time the job is running/completed and fal's
cancel envelope is {detail:...} (no status field). The leg hard-asserted
`typeof cancel.body.status === 'string'` -> AssertionError -> collector
exit-5 quarantine -> the drift-live-pr base step red on every PR (descoped in
#330).

Fix (probe-only, no fal.ts change):
- Lifecycle: state-agnostic submit -> status -> CANCEL; accept any valid
  queued/running/completed status; cancel always fires even if status polls
  throw. Relax the brittle cancel assertion to require only a JSON envelope
  ({status} OR {detail}); leave shape grading to triangulate so a genuine
  2xx-wrong-shape still reports critical drift.
- Infra-skip (folds in #329, which is being closed): 401/402/403 locked-or-
  exhausted/429/5xx raise a distinguished FalCanarySkip -> honest ctx.skip,
  never an InfraError that quarantines the baseline.
- Re-enable: drop the FAL_LIVE_QUEUE opt-in gate #330 added so the corrected
  leg runs live in CI again (still gated on FAL_KEY).

Cost-safe: cheapest model, cancel ASAP, result endpoint never fetched.
Residual: <= one sub-cent generation if the model races to completion.

Red-green in fal-canary-skip.test.ts drives the real canary + real triangulate:
IN_PROGRESS+{detail} handled (leg passes), 403 -> FalCanarySkip, 2xx-wrong-
shape still critical.
jpr5 added a commit that referenced this pull request Jul 23, 2026
@jpr5
fix: fal probe handles IN_PROGRESS + infra-skip; re-enable live queue…
b567191 
… leg

The live fal queue-lifecycle canary (#327) assumed jobs sit in IN_QUEUE long
enough to cancel for $0. Funded flux/schnell returns status:"IN_PROGRESS"
on submit immediately; by cancel time the job is running/completed and fal's
cancel envelope is {detail:...} (no status field). The leg hard-asserted
`typeof cancel.body.status === 'string'` -> AssertionError -> collector
exit-5 quarantine -> the drift-live-pr base step red on every PR (descoped in
#330).

Fix (probe-only, no fal.ts change):
- Lifecycle: state-agnostic submit -> status -> CANCEL; accept any valid
  queued/running/completed status; cancel always fires even if status polls
  throw. Relax the brittle cancel assertion to require only a JSON envelope
  ({status} OR {detail}); leave shape grading to triangulate so a genuine
  2xx-wrong-shape still reports critical drift.
- Infra-skip (folds in #329, which is being closed): 401/402/403 locked-or-
  exhausted/429/5xx raise a distinguished FalCanarySkip -> honest ctx.skip,
  never an InfraError that quarantines the baseline.
- Re-enable: drop the FAL_LIVE_QUEUE opt-in gate #330 added so the corrected
  leg runs live in CI again (still gated on FAL_KEY).

Cost-safe: cheapest model, cancel ASAP, result endpoint never fetched.
Residual: <= one sub-cent generation if the model races to completion.

Red-green in fal-canary-skip.test.ts drives the real canary + real triangulate:
IN_PROGRESS+{detail} handled (leg passes), 403 -> FalCanarySkip, 2xx-wrong-
shape still critical.
jpr5 added a commit that referenced this pull request Jul 23, 2026
@jpr5
fix: fal probe handles IN_PROGRESS + infra-skip; re-enable live queue…
3565ec4 
… leg (#332)

## What

Proper fix for the fal live queue-lifecycle drift probe
(`falQueueLifecycleCanary` + the live block in
`src/__tests__/drift/fal-queue.drift.ts`, merged #327), and
**re-enables** the live leg that #330 temporarily descoped. Probe-only —
no change to aimock's `fal.ts` core.

## The bug (verified live in CI, root cause)

The probe assumed fal jobs sit in `IN_QUEUE` long enough to cancel for
\$0. Funded `flux/schnell` returns `status:"IN_PROGRESS"` on the submit
envelope **immediately**. By cancel time the job is running/completed,
so fal's cancel envelope is `{detail:"..."}` — **no `status` field**.
The live leg hard-asserted `typeof cancel.body.status === "string"` →
AssertionError → the collector could not map it → **exit-5 quarantine**
→ the `drift-live-pr` **base** step (main's code) went red on every PR.
#330 was the fast, reversible descope; this is the proper fix.

## The fix

1. **Lifecycle** — state-agnostic submit → status → CANCEL. Accept any
2xx submit and any valid queued/running/completed status (`IN_QUEUE`
**or** `IN_PROGRESS` …), never assert the job is specifically queued.
Cancel still fires even if the status poll throws (a job is never left
running).
2. **Envelope shapes / drift** — the cancel assertion now requires only
a JSON envelope (`{status}` **or** `{detail}`); precise shape grading is
left to `triangulate` against the mock, so a genuine **2xx-wrong-shape
still reports critical drift**.
3. **Infra-skip (folds in #329, which is being closed in favor of this
PR)** — `401/402/403` (locked/exhausted) `/429/5xx` raise a
distinguished `FalCanarySkip` → honest `ctx.skip`, never an `InfraError`
that quarantines the baseline. Includes #329's `fal-canary-skip.test.ts`
guard tests.
4. **Re-enable** — removes the `FAL_LIVE_QUEUE` opt-in gate #330 added;
the corrected leg runs live in CI again (still gated on `FAL_KEY`).

## Cost safety

Cheapest reliably-available model (`fal-ai/flux/schnell`), cancel ASAP,
the paid `response_url` result endpoint is **never** fetched. Residual:
at most **one sub-cent** generation if the model races to completion
before the immediate cancel lands.

## Red-green proof (local, drives the real canary + real `triangulate`)

RED captured against pre-fix probe (origin/main @ d43701c):

```
× RED(a): IN_PROGRESS submit + {detail} cancel — old assert `typeof cancel.body.status === "string"`
    → AssertionError: expected 'undefined' to be 'string'
× RED(b): 403 user-locked at submit
    → rejects with InfraError: "INFRA_ERROR: fal Queue Lifecycle …" (quarantine-causing, not a skip)
```

GREEN post-fix (`src/__tests__/drift/fal-canary-skip.test.ts`, 4/4
pass):
- (a) IN_PROGRESS submit + `{detail}` cancel → canary handles it,
`triangulate` reports no critical → leg passes.
- (b) 403 user-locked → `FalCanarySkip` (not `InfraError`); 401/402 also
skip.
- (c) **guard**: 2xx submit whose envelope diverges from the mock →
still critical drift (fix did not swallow genuine drift).

## Checks (local)

`prettier` ✓ · `eslint` ✓ · `tsc --noEmit` (tests-inclusive) 0 errors ✓
· `test:drift` 16 passed / 9 skipped ✓ · full suite 4693 passed ✓ ·
`tsdown` build ✓

Live validation (corrected leg against funded fal) = CI `drift-live-pr`
on this PR — see checks below.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01KamK73Wu5heLxJEogM6hHC
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@jpr5