Repository navigation
pm-dispatch: POST /v1/sessions and /v1/tasks return HTTP 502 (2026-11-04) #5647
Description
Problem
pm-dispatch (https://dispatch.security.cab) returned HTTP 502 for both dispatch endpoints on 2026-11-04, verified twice from GitHub Actions runs of developing-today/opencode's Daily Upstream Sync workflow after PR developing-today-forks/opencode#1671 merged:
- Run 41902737337 (2026-11-04T20:02Z, head e4abd8c3):
pm-dispatch-start: pm-dispatch POST /v1/sessions failed (HTTP 502); the failure-handoff step then also failed:pm-dispatch-queue: pm-dispatch POST /v1/tasks failed (HTTP 502). - Run 41903121391 (2026-11-04T20:05Z): identical HTTP 502 on both
POST /v1/sessionsandPOST /v1/tasks.
Both runs got past job setup and action loading (the #4742 _staging defect those runs were verifying is fixed), minted the OIDC token, and reached the dispatch service — the 502 is from the service / its gateway, not the caller.
Impact
Every workflow that launches or queues work via pm-dispatch fails until the service recovers, and the failure-handoff path itself depends on the same service, so failures during an outage are not queued either.
Acceptance
-
POST /v1/sessionsandPOST /v1/tasksfrom a GitHub Actions OIDC caller return 2xx again (a dispatched Daily Upstream Sync run completing its dispatch step is sufficient evidence). - Root cause of the 2026-11-04 ~20:02–20:05Z 502s identified and noted here.
Filed by queue task 7519 (opencode #1519 verification) — that task is parked on this outage; its own staging fix is merged and proven.
Acceptance criteria
POST /v1/sessionsandPOST /v1/tasksfrom a GitHub Actions OIDC caller return 2xx again (a dispatched Daily Upstream Sync run completing its dispatch step is sufficient evidence).- Root cause of the 2026-11-04 ~20:02-20:05Z HTTP 502s is identified and recorded on this issue.
- The failure-handoff path's dependence on the same service is dispositioned (documented as accepted risk, or a fallback is filed as a follow-up issue).
Activity
developing-today commentedon Nov 4, 2026
CODE-1406
dezren39 commentedon Nov 4, 2026
Intake review
## Acceptancechecklist is preserved above; the appended section restates it in the fleet's standard form.— fleet monitor intake, 2026-11-04
9 remaining items
dezren39 commentedon Nov 5, 2026
🤖 Agent note · parked (task 10651, lane code-s)
Root-cause fix opened as PR #6324: the pm-tunnel unit (
nixos/services/pm-stack/default.nix) ran cloudflared with no--protocolflag — default QUIC — and the tunnel's QUIC connections are what drop on this host (no recent network activity, 2026-11-05T11:36Z, shortly before run 42023264697's edge 502s). PR #6324 switches the tunnel to--protocol http2(TCP); flag placement verified against the deployed cloudflared 2026.10.7 binary.Resume condition: merge PR #6324,
nixos-rebuild switchon the laptop, confirm the tunnel registers on http2 with no QUIC drop errors, then verify a real OIDC dispatch / Daily Upstream Sync run completesPOST /v1/sessionswith 2xx — only that run-level evidence closes this issue. Parked rather than held: the deploy restarts the fleet's tunnel and should ride a review of PR #6324 first.developing-today commentedon Nov 5, 2026
🤖 Agent note · picked up (resume)
--protocol http2to the pm-tunnel ExecStart exactly as scoped; Codex review on head b5df1bb completed with no findings; no unresolved review comments. Merging PR fix(pm-stack): run the pm Cloudflare tunnel over HTTP/2, not QUIC (#5647) #6324, then deploying via nixos-rebuild switch on the laptop and verifying the tunnel on http2.Merge PR #6324: pm-tunnel over HTTP/2 (#5647)
developing-today commentedon Nov 5, 2026
🤖 Agent note · parked — merged, deploy blocked on interactive sudo (task 10651, lane code-k)
Done this pass: PR #6324 merged — merge commit
1188e6ad24252d90e6823664f3704c6a6a5ddebe(verified fresh after merge). Laptop~/git/codeis already at that commit (pm-git-sync), and the mergednixos/services/pm-stack/default.nixcarries--protocol http2in the pm-tunnel ExecStart.Blocker: the running cloudflared (user unit
pm-tunnel.service, home-manager managed) is still the pre-fix QUIC process. Activating the fix needssudo nixos-rebuild switch --flake /home/user/code#amdon the laptop, and fleet sessions have no non-interactive sudo there (sudo: a password is required, no NOPASSWD rule, no standalone homeConfigurations output in the flake). A person at the laptop (or a root session) runs that one command; the tunnel unit restarts on HTTP/2.Resume condition (after the rebuild): confirm
journalctl --user -u pm-tunnelshows tunnel connections registered with protocol http2 and no new QUIC drop errors; then run-level evidence per the acceptance criteria (OIDC dispatch or next Daily Upstream Sync completes POST /v1/sessions with 2xx) — #5647 closes only on that evidence.