A bundled PostgreSQL pod restart leaves ate-api-server NotReady and failing for 2 minutes: pgx dials the stale pod IP with no connect timeout #2995

Summary

Deleting the bundled postgres-0 pod (it was back and Running 2/2 within 20 s) left both ate-api-server replicas NotReady and failing every store call for about 2 minutes. The cause is visible in the logs: ate-api-server kept dialing the old pod IP and each attempt only failed after the OS TCP connect timeout, because the pgx pool has no connect timeout configured. Clients saw codes.Internal for the whole window.

Environment

• Substrate main at 4b4fbb2 (2026-10-20), ate-setup deploy ate-system defaults: bundled PostgreSQL StatefulSet behind the headless Service postgres (clusterIP: None), 2 ate-api-server replicas.
• GKE 1.36.4, 3x n2-standard-8. Light steady load: 20 glutton users, async suspend (pause) and resume, about 12 requests/s through the ateapi gRPC API.

Reproduction

kubectl -n ate-system delete pod postgres-0 --grace-period=0 --force
watch kubectl -n ate-system get pods -l app=ate-api-server     # READY 1/2 on both replicas for ~2 min

Timeline (Cloud Logging, ate-api-server container, UTC)

Client side (Locust/boomer over the ateapi Service): ResumeActor 314 failures out of 552 in the window, all code = Internal desc = internal server error: getting actor <name>: failed to connect ... or ... workflow failed at step FinalizeRunning: getting actor ...; one call blocked for 127 s. One response carried rpc error: code = Code(4000) desc = Internal Server Error inside the Internal text (the authorization store client's error).

Actors: 20 of 22 ended SUSPENDED, no CRASHED, no stuck transitional state; the invariant checker was clean two minutes later.

Analysis

• The StatefulSet pod comes back with a different IP behind a headless Service. Connections resolved to the old address have to fail before the pool reconnects.
• cmd/ateapi/internal/store/atepg/atepg.go poolConfig sets no ConnConfig.ConnectTimeout and the DSN carries no connect_timeout, so a dial to a dead address runs for the Linux SYN retry budget (about 127 s, which matches 19:25:40 to 19:27:40 exactly). The readiness probe's store check shares the same pool configuration, so /readyz stays 503 for the same period and the whole control plane is unavailable for 2 minutes after a 20 second database restart.
• The error reaches clients as Internal (non-retriable for the router's parking logic and for kubectl-ate), not Unavailable. Support OCI seccomp profiles inside gVisor sandboxes #2277 covers hiding the raw pgx text; it does not change the code or the recovery time.

What would help

• Set a short connect timeout on the pgx pool (ConnConfig.ConnectTimeout of a few seconds, or connect_timeout in the DSN) so a stale address fails fast and the pool re-resolves.
• Classify store connectivity failures as Unavailable so callers and the router retry instead of failing hard.
• Document that a bundled PostgreSQL restart is expected to cause a brief outage and how long.

Evidence

Cloud Logging export of the window (690 entries), readiness probe events, Locust stats and the invariant checker output are saved from the run and available on request.