[Bug]: Workers can't connect to atelet's worker-support socket on dense nodes ("connect: resource temporarily unavailable") #3575

What happened?

On dense nodes running personal-assistant-light (~5,000–6,000 actors on c3-highcpu-192-metal), calls from ateworker to atelet over /var/lib/ate/worker-support.sock fail due to socket backlog exhaustion.

Across our benchmark runs, worker pods consistently log thousands of connection failures:

• US node:
  • 1,283 × dial unix /var/lib/ate/worker-support.sock: connect: resource temporarily unavailable (EAGAIN)
  • 1,335 × rpc error: code = DeadlineExceeded desc = context deadline exceeded while waiting for connections to become ready
• EU node:
  • 794 × connect: resource temporarily unavailable
  • 1,678 × DeadlineExceeded

Expected Behavior

ateworker can reliably query atelet over /var/lib/ate/worker-support.sock to fetch and refresh actor egress policies without socket backlog exhaustion or connection drops, even under high actor density (e.g. 5,000+ actors on large metal nodes).

Steps to Reproduce

1. Deploy Agent Substrate on a large bare-metal node (e.g. c3-highcpu-192-metal with 192 vCPU, 503 GiB RAM) running 8 worker pods.
2. Run a high-density actor workload (e.g. personal-assistant-light / LPA at 250m CPU / 512 MiB memory).
3. Scale the concurrent actor load past 5,000 actors.
4. Inspect ateworker pod logs: observe massive floods of dial unix /var/lib/ate/worker-support.sock: connect: resource temporarily unavailable (EAGAIN) and connection DeadlineExceeded errors.

Sandbox Runtime

gVisor (runsc)

Agent Substrate Version / Commit SHA

475f49b

Kubernetes Version & Environment

GKE on c3-highcpu-192-metal

Host OS & Architecture

Linux 6.6 (x86_64)

Relevant Logs and Diagnostic Output

Worker error logs when attempting to refresh egress policies over the UNIX socket:
ERROR Could not refresh the actor's egress policy; using stale cached policy while fetching the actor's egress policy: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /var/lib/ate/worker-support.sock: connect: resource temporarily unavailable"

Subsequent connection timeouts:
ERROR Could not refresh the actor's egress policy; using stale cached policy while fetching the actor's egress policy: rpc error: code = DeadlineExceeded desc = context deadline exceeded while waiting for connections to become ready

New actor startup failures:
ERROR Could not fetch the actor's egress policy; refusing its DNS queries err: dial unix /var/lib/ate/worker-support.sock: connect: resource temporarily unavailable

Additional Context

In internal/atunnel/egresspolicy.go, each active actor polls atelet to refresh its egress policy every 5 seconds (egressPolicyTTL):

// egressPolicyTTL is how long an actor's egress policy is reused before it
// is fetched again.
//
// TODO: tune once the cost of fetching is known at scale.
egressPolicyTTL = 5 * time.Second

At 6,000 actors per node, polling every 5s produces a minimum steady-state load of 1,200 RPCs/sec over the single /var/lib/ate/worker-support.sock Unix domain socket. Under load, this blows past atelet's listener accept backlog, causing EAGAIN on connect().

Across multiple test runs on c3-highcpu-192-metal:

• US node: 1,283 × resource temporarily unavailable + 1,335 × DeadlineExceeded
• EU node: 794 × resource temporarily unavailable + 1,678 × DeadlineExceeded

Impact:

1. Running actors fall back to stale cached policies with heavy ERROR log spam.
2. If an actor's initial policy fetch fails during startup, it logs "Could not fetch the actor's egress policy; refusing its DNS queries" (internal/atunnel/egresspolicy.go:145-151), causing new actors to fail startup and liveness checks.
3. The same socket carries other node RPCs (e.g. crash reporting and checkpoints), so saturating the socket listen queue degrades other operations on the node.

Potential fixes to consider:

• Tuning egressPolicyTTL (as noted in the author's TODO, 5s is too aggressive for dense nodes).
• Increasing the Unix domain socket listen backlog in atelet.
• Batching/coalescing policy checks per worker rather than having every actor poll individually.
• Push-based invalidations from atelet on policy change instead of continuous polling.

Confirmation

• I have searched existing issues and verified that this is not a duplicate.
• I have verified that this issue occurs on the latest commit on main.