Skip to content

Egress gateway: stop timing out ext_proc calls and policy fetches #2209

Open

Description

Problem

The egress gateway puts fixed timeouts on its path to ateapi:

  • Envoy's ext_proc message_timeout is 5s on all three egress filters in manifests/ate-install/atenet-egress.yaml, and grpc_service.timeout is 2s on the CONNECT leg and 5s on the others.
  • The policy cache gives up on GetActorEgressPolicy after 4s (policyFetchTimeout).

When ateapi slows past these limits, every request that needs a fetch fails, either as Envoy's 504 or as the gateway's 503. A fetch that would have finished a little later is thrown away instead of filling the cache, so the next request fails the same way. A per-call timeout doesn't shed load; it turns a slow control plane into an outage (review on #2135).

Proposal

  • Raise message_timeout on the egress ext_proc filters to Envoy's maximum of 1h. Leaving it unset would mean 200ms, not no limit. Raise grpc_service.timeout to match so it can't become the cap.
  • Drop policyFetchTimeout. The fetch already outlives the request that started it, so a slow fetch would land in the cache for the next request.
  • Extend TestEgressManifestsExtProcFilters so the timeouts can't quietly shrink again.

If the gateway needs to shed load, it should turn new requests away up front. Today that's Envoy's default circuit breaker on ext_proc_server, which allows 1024 concurrent requests.

#2204 has a draft of this change.

Open questions

  • Is 1h right, or should the limit match ateapi's 10-minute server-side RPC cap?
  • Should the ingress router's ext_proc timeouts, which are tied to request parking, follow the same rule?

Related: #2135, #2204.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions