Repository navigation
Review global-401/429 guard thresholds once real log data exists #7
Description
Context
The global-outage guards (added in #5) use fixed thresholds tuned for the
minimum pool size, with no production data behind them:
global401WindowMs= 30s, trigger = ≥ 2 DISTINCT accounts 401ing- same window for 429, trigger = ≥ 2 distinct accounts
- escape hatch: suppression lifts after more than 1 rejection per account
within the window
Why this needs real data
With exactly 2 accounts, two genuine per-account failures in 30s look
identical to an upstream-wide incident. The suppression heuristic
(repeated bursts → allow marking) mitigates the stuck-forever case, but
the threshold itself is a guess.
What to review (after ~1 week of production logs)
- Unknown-error log lines (status + redacted headers + body preview)
accumulated since deploy — do real 429s carry distinguishing
headers/biz codes that beat the timing heuristic? - Did any global-401/429 suspicion fire? If yes: was it actually global
(both accounts were fine later) or per-account (both were dead)? - Is 30s the right window for Claude Code traffic patterns (bursts of
subagent requests)?
Inputs available
- Unknown-error evidence logging (from Tích hợp AccountPool vào proxy handler (failover-ready routing) #4/Failover theo phân loại lỗi (retry trước headers, capture mẫu lỗi thật) #5) is always on
/quotashows per-accountlastPolledAt/ status history- The poller can act as an independent probe to disambiguate (poll
success = credential alive)
docs/multi-account.md documents the current thresholds.
Activity
longphamdev commentedon Nov 9, 2026
Dữ liệu production đầu tiên (E2E 2 acc thật, 2026-11-09):
Setup: acc1 = start-plan hết quota (biz 1005), acc2 = start-plan mới login, cả hai cùng IP.
Kết quả:
biz-1005 classification hoạt động production:
pool: account <id> quota exhausted (biz 1005)— matcher theo body OK, không miss.Failover thật đầu tiên: request → acc1 (1005) → exhausted → rotate → acc2 → 200 OK. Body replayable, client không thấy lỗi. Sau đó 5 request kế đi thẳng acc2 (exhausted được nhớ) — không đốt 1005 lặp lại. Đúng thiết kế.
429 thật captured (điều ta chưa từng có mẫu): 6 concurrent requests → 3× 200, 1× upstream 429 trên acc2, 2× 503 (all-busy khi hit cap + acc1 exhausted). acc2 bị rate limit với chỉ 3 stream đồng thời — acc free rate limit rất thấp.
Quan sát cần review (lý do issue này tồn tại):
coolAlltrong lúc acc1 đã exhausted sẵn → hiệu ứng "cả pool khóa" dễ xảy ra khi pool = 2 và 1 đã dead.Đề xuất xem lại:
Retry-Aftergiá trị thật vào cooling line để đo đạc.Verify khẳng định hoạt động: no leaked slots (mọi request xong inFlight về 0), không watchdog warning, stream dài chưa test (cần dùng thật thêm).
fix(pool): 429 guard correctness — usable-based parity, terminal prec…
longphamdev commentedon Nov 9, 2026
Follow-up on the live findings (fixes deployed):
Poll-0 vs successful requests — implemented the ground-truth rule: a
successful REQUEST invalidates a stale
remainingUnits: 0poll reading(pool clears it on noteSuccess). The poller itself never marks exhausted
from remaining=0 (verified — it only feeds data + heals).
/quotanowflags a healthy account reading 0 as
remainingUnreliable: trueinsteadof presenting 0 as fact.
Global-429 guard fix — usableCount() (healthy/cooling) now drives
single-acc parity AND the global guard; exhausted/expired accounts are
excluded. With a 2-acc pool where one is dead, the live account behaves
as single-acc (pass-through parity) instead of coolAll-ing a pool with
nowhere to rotate. Terminal precedence: cool() is a no-op on
exhausted/expired — a stray 429 on a half-open probe cannot resurrect the
account early.
Internal cooling clamp — pool cooling is capped at 6h even when the
upstream Retry-After is huge (weird header must not lock an account out
for days). The client still sees the verbatim header.
maxInFlightPerAccount lowered 4 → 2 — free-tier 429'd with ~3
concurrent streams; the 5s queue now absorbs bursts instead of feeding
the upstream limit. Will watch AccountsBusyError rate.
Measured so far: upstream 429 hit acc2 with 3 concurrent streams
(Retry-After value not yet captured — the passthrough path was dropping the
header; now forwarded verbatim on both response paths, so next occurrence
will be measurable).
fix(pool): request outranks balance poll; cooling clamp; remainingUnr…
fix(poller): per-model balance buckets — snake_case fields, real quot…
feat(quota): per-model buckets in /quota — remove misleading cross-mo…
feat(config): accounts.pickStrategy — switchable pick strategy with s…