[Bug]: DNS queries each wait up to 2 s for a new egress-policy fetch after a refresh fails #3738

What happened?

An actor's DNS queries get its cached egress policy when a refresh fails. Each query still waits for a new fetch first, up to egressPolicyFetchTimeout (2 s) when atelet or ate-api-server hangs. A failed refresh never moves expires (internal/atunnel/egresspolicy.go:153-165 at 94e0a26a). Only a successful fetch sets it (:180).

The relay reads an actor's UDP queries in one loop (internal/atunnel/dns/udphandler.go:119-140). Queries queued behind a waiting one wait too, and each then starts another fetch.

Expected Behavior

Queries get the stale policy without waiting while refreshes fail. actorEgressPolicy retries the fetch at most once per TTL.

Steps to Reproduce

Unit test against main (94e0a26a). I haven't reproduced it on a cluster.

1. Save the test below as internal/atunnel/stale_wait_test.go. It caches a policy, then makes every later fetch hang until its timeout.
2. On Linux, run go test -v -run TestStalePolicyQueriesWaitForRefresh ./internal/atunnel/.

Each query returns the stale policy after 2 s and starts a new fetch.

//go:build linux

package atunnel

import (
	"context"
	"sync/atomic"
	"testing"
	"time"

	"go.ate.dev/substrate/internal/proto/ateletpb"
)

func TestStalePolicyQueriesWaitForRefresh(t *testing.T) {
	policy := &ateletpb.EgressPolicy{}
	var fetches atomic.Int32
	source, clock := newTestSource(func(ctx context.Context) (*ateletpb.EgressPolicy, error) {
		if fetches.Add(1) == 1 {
			return policy, nil
		}
		<-ctx.Done() // atelet or ate-api-server hangs
		return nil, ctx.Err()
	})
	if _, err := source.EgressPolicy(context.Background()); err != nil {
		t.Fatal(err)
	}
	clock.advance(egressPolicyTTL)

	start := time.Now()
	for i := range 3 {
		queryStart := time.Now()
		got, err := source.EgressPolicy(context.Background())
		if err != nil || got != policy {
			t.Fatalf("query %d: got (%v, %v), want the stale policy", i, got, err)
		}
		t.Logf("query %d answered with the stale policy after %s", i, time.Since(queryStart).Round(time.Millisecond))
	}
	refetches := fetches.Load() - 1
	t.Logf("3 queries took %s and started %d refetches", time.Since(start).Round(time.Millisecond), refetches)
	if refetches > 1 {
		t.Errorf("got %d refetches within one TTL, want 1", refetches)
	}
}

Sandbox Runtime

Both / Runtime Agnostic

Agent Substrate Version / Commit SHA

94e0a26

Kubernetes Version & Environment

Not applicable (unit test).

Host OS & Architecture

Linux 6.6 (x86_64), Go 1.27.3

Relevant Logs and Diagnostic Output

stale_wait_test.go:36: query 0 answered with the stale policy after 2.001s
    stale_wait_test.go:36: query 1 answered with the stale policy after 2.001s
    stale_wait_test.go:36: query 2 answered with the stale policy after 2s
    stale_wait_test.go:39: 3 queries took 6.004s and started 3 refetches
    stale_wait_test.go:41: got 3 refetches within one TTL, want 1
--- FAIL: TestStalePolicyQueriesWaitForRefresh (6.00s)

Additional Context

#3575 shows refreshes timing out with DeadlineExceeded on dense nodes. #3577 should make these failures rarer. It doesn't change what a failed refresh costs each query. #2209 covers how the egress gateway's policy cache handles a slow ate-api-server.

I have a fix ready and can send it as a PR if the approach works for you. After a failed refresh, it keeps the stale entry for another TTL before retrying. While refreshes fail, queries get the stale entry and the next refresh runs in the background. Healthy refreshes and the first fetch behave as today.

The fix can serve the stale policy up to about one TTL (5 s) longer after atelet recovers. T-26 in docs/security/threat-model.md says an enforcement point "denies rather than serving stale policy indefinitely". The DNS relay's readme (internal/atunnel/dns/readme.md) documents serving it while atelet is temporarily unreachable. Should stale policy have a maximum age, after which queries are refused?

AI use: Claude-assisted code tracing. I checked the code and the test output myself.

Confirmation

• I have searched existing issues and verified that this is not a duplicate.
• I have verified that this issue occurs on the latest commit on main.