[P2] No fault-injection e2e tests for any critical failure scenario #614

Severity: P2 (all P0/P1 issues invisible to CI; regressions will reach production)
Component: Test infrastructure — internal/e2e/
Audit ID: NET-10

Summary

The e2e test suite covers happy-path lifecycle, identity after restore, network policy,
metrics emission, and request parking. There are no tests for any of the critical
failure scenarios identified in this audit: node deletion after pause (ISSUE-008),
ateapi crash mid-workflow (ISSUE-010), cold-actor first-request failure (ISSUE-009),
checkpoint re-invocation on torn-down workload (ISSUE-004), or Valkey data loss
(ISSUE-001). The longest test in the suite runs for ~10 seconds — the 30-second lock
TTL and 5-second ext_proc timeout boundary are never exercised. Every P0 and P1 issue
in this audit would have been caught immediately by a fault-injection test.

What Exists

File: internal/e2e/suites/ (confirmed directories):

• demo/ — happy path lifecycle, actor routing, pod deletion (TestWorkerPodDeletion)
• identity/ — identity cert after restore
• networkpolicy/ — network isolation
• metrics/ — OTel metric emission
• parking/ — BudgetExhaustion, ParkThenServed sub-tests (~10s total)

No suite covers:

• Component crash + recovery (ateapi, atelet, atenet-router)
• Valkey restart or data loss
• Object store outage (GCS/S3 unavailable during checkpoint)
• PAUSED actor + node deletion
• Concurrent lifecycle RPCs on same actor
• Disk full on atelet node

Impact

• P0/P1 regressions (ISSUE-001 through ISSUE-010) are undetectable in CI.
• A developer fixing one issue could re-introduce another with zero automated feedback.
• The stability plan (SUBSTRATE-STABILITY-PLAN.md) requires manual execution of all
  experiments because no automated harness exists.

Steps to Demonstrate the Gap

# Run the full e2e suite — observe zero fault-injection coverage
hack/run-e2e-kind.sh  # should pass cleanly
grep -rn "kill\|crash\|fault\|inject\|SIGKILL\|SIGSTOP\|rollout restart" internal/e2e/
# Returns: zero results

Expected Behavior

A fault-injection test suite in internal/e2e/suites/stability/ covering:

Suggested Fix

1. Add stability/ test suite under internal/e2e/suites/.
2. Use kubectl delete pod + kubectl wait for component kill injection (no external
   chaos framework needed — this matches the stability/faults/ script approach in
   SUBSTRATE-STABILITY-PLAN.md).
3. Add an invariant checker (per §2.1 of SUBSTRATE-STABILITY-PLAN.md) as a test
   helper that runs after each fault scenario.
4. Gate these tests with a build tag //go:build stability so they run in nightly CI
   but not in every presubmit.