Repository navigation
clef-proxy: data-path upstream fetch has no timeout — a gateway that accepts and never responds holds the /v1/systemone request open (models fetch in the same file pins 10s) #2100
Description
clef-proxy: data-path upstream fetch has no timeout — a gateway that accepts and never responds holds the /v1/systemone request open (the models fetch in the same file pins 10s)
Evidence
bin/clef-proxy.mjs (main 82b019a0, sha256 12e3415ff346ac25…):
:234— the data path, the proxy's only inference call, awaits the Cloudflare gateway with no signal and no deadline of any kind:
const cfRes = await fetch(cfUrl, {
method: 'POST',
headers: { 'Authorization': `Bearer ${token}`, 'Content-Type': 'application/json' },
body: JSON.stringify(payload)
});cfUrl is built at :227 (https://gateway.ai.cloudflare.com/v1/<account>/<gateway>/workers-ai/run/<model>). The follow-up await cfRes.json() at :248 is likewise unbounded, so a gateway that sends headers and then trickles or stalls the body hangs the same handler a second way.
-
:128-130— the same file already knows the fix pattern: the live-model-catalog fetch pinssignal: AbortSignal.timeout(10000). Only the catalog got a deadline; the request that actually serves user traffic did not. -
The handler is the
POST /v1/systemoneroute: until the fetch settles it never writes a response, so the client connection, the request handler, and (for the hosted SystemOne caller) the turn all stay open. Nothing maps a stall to a 504 — if the runtime eventually errors the fetch, the shared catch at the end of the handler reports it as a generic 500.
Probe (this run, sandbox Node, verbatim call shapes)
A local HTTP server that accepts and never responds, called two ways:
| call | result |
|---|---|
fetch(url, { signal: AbortSignal.timeout(2000) }) (control) |
rejected after ~2000 ms (TimeoutError) |
fetch(url, { method: 'POST', headers, body }) — exactly the :234 shape, no signal |
still unsettled after 12 s, probe exited with the request pending |
So the data path's only bound is whatever the fetch runtime's global defaults happen to be (undici's multi-minute headers/body defaults) — roughly 30× the 10 s deadline this same file chose for its catalog fetch, and not a contract anything pins or tests.
Why it matters
Clef is a decision endpoint: a stalled gateway turns into a stalled caller with no error, no log line naming a timeout, and no response to retry against, for minutes at a time. The proxy process stays healthy (/health answers), so supervision sees nothing. This is the same failure class as #2097 (design-code-proxy dispatch, no timeout, fallback never fires) in a different proxy whose upstream is a hardcoded public gateway — there is no operator-side workaround short of editing the file.
Acceptance criteria
- The data-path fetch at
:234carries an explicit deadline (e.g.AbortSignal.timeout(<pinned value>)), and the deadline also bounds reading the response body (cfRes.json()). - A deadline expiry is reported as an upstream timeout (a 504-class response naming the timeout), not folded into the generic 500 catch.
- The pinned value is a named constant (parity with the models fetch's 10000, or a triage-pinned Clef-appropriate value), with the choice recorded on this issue.
- Tests in
bin/clef-proxy.test.mjscover: a never-responding upstream fails the request at the deadline with the timeout response, and a normally responding upstream is unchanged. - The models-catalog fetch behaviour is unchanged.
Related: #2097 (same no-timeout class, design-code-proxy), #2096 / #1809 (this proxy family's env-parsing cases — different failure, same "starts healthy, fails silently" shape).
Activity
developing-today commentedon Oct 29, 2026
CODE-1009
dezren39 commentedon Oct 29, 2026
🤖 Agent note · filed
18 remaining items
dezren39 commentedon Nov 1, 2026
🤖 Agent note · handed off
node --test bin/clef-proxy.test.mjspasses 15/15 but does not cover a hanging upstream. The issue remains open and no active related PR was found.dezren39 commentedon Nov 2, 2026
🤖 Agent note · picked up — OpenChamber scheduled lane
ses_e5e664d0fddffeHjJoK3HnMEFmrepo-health-code-pilotdezren39 commentedon Nov 2, 2026
🤖 Agent note · done — scoped data-path timeout PR opened
pr_opened; PR fix(clef-proxy): bound data-path gateway requests #4096 fix(clef-proxy): bound data-path gateway requests #4096node --test bin/clef-proxy.test.mjspassed 28/28;git diff --checkpassed.gh pr checksshowed only CodeRabbit skipping; no CI checks were active.queue/5184-2100-data-path-timeoutatb3afe605a0a0c6e1a9d31088ca13ef5609b59b29. Queue in-flight labels removed; sandbox backlog/sweep bookkeeping deferred to fleet reconciliation.