Repository navigation
id(world_net): inbound_refusal_backpressures test is flaky — intermittent "a pre-auth refusal sheds the stream" #5195
Description
Observed 2026-11-03 on the laptop (amd-6) while verifying the #5020/#5021/#5128/#5132 fixes — unrelated to those changes (different subsystem; none of them touch world_net), filed so it exists as a tracked problem:
world_net::tests::inbound_refusal_backpressures_authorized_streams_but_sheds_preauth panics intermittently at src/world_net.rs:2171:
a pre-auth refusal sheds the stream
Evidence, all on origin/main 80b06f8:
- Full
cargo test -p idrun under parallel load: lib target 2038 passed / 1 failed — this test. - In isolation (
cargo test -p id --lib -- inbound_refusal_backpressures), three consecutive runs: pass, fail, pass. - Same day, a full
--no-fail-fastrun (43 targets, 2501 passed / 0 failed) and thenix build .#id-websandbox check phase (3294 passed / 0 failed) both had it green.
So it is timing/load-sensitive rather than broken. It still matters: .#id-web runs this suite in its Nix check phase (doCheck=true), so an unlucky run can intermittently redden the build and block the laptop fleet switch with a failure that is not a regression.
Acceptance: the test passes reliably under load (e.g. repeated full-lib runs on a busy machine), by fixing the synchronization/timing assumption in the test (or the backpressure behavior, if the flake exposes a real race).
Acceptance criteria
world_net::tests::inbound_refusal_backpressures_authorized_streams_but_sheds_preauthpasses reliably under load: repeated fullcargo test -p id --libruns on a busy machine (10+ consecutive) with zero failures of this test.- The fix addresses the synchronization/timing assumption in the test — or, if the flake exposes a real backpressure race, the behavior in
src/world_net.rs— with the root cause named in the closing comment. - The
nix build .#id-webcheck phase stays green.
Activity
dezren39 commentedon Nov 3, 2026
Intake review
Related: Filed while verifying the #5020/#5021/#5128/#5132 fixes (different subsystem). CI-runner context: #5194.
Notes: Flaky-test filing with strong evidence: pass/fail/pass in isolation on main 80b06f8, green in two other full runs the same day. No merged PR touches world_net backpressure in a way that resolves this (merged-PR search this run: none). Risks: timing flakes invite timeout-bump non-fixes; the acceptance bar (10 consecutive full-lib runs under load) is set to reject those. Suggested approach: reproduce under load first, inspect the await/poll assumptions around src/world_net.rs:2171, prefer a deterministic synchronization fix over sleeps.
— fleet monitor intake (protocol §5), 2026-11-03. Verified still live this run: no merged PR resolves this issue.
id(world_net): make inbound_refusal_backpressures test deterministic (#…
Merge pull request #5215 from developing-today/fix/5195-inbound-refus…
dezren39 commentedon Nov 4, 2026
Fixed in #5215 (merge
03755976447ef02030efc66f584b67fc9befa714on main).Root cause
A timing assumption in the test, not a backpressure race. The test configured
frames_per_sec: 1_000, frame_burst: 1(and the same for bytes). The evaluator buckets (src/abuse.rs) count milli-tokens: capacity 1_000, refill 1_000 milli-tokens per millisecond. After the first pre-auth charge drains the bucket, one full token is back after 1 ms of wall time — so the shed assertion ("second pre-auth charge is refused") only holds if the test thread happens to run the second charge within that same ~1 ms window. Any scheduling stall ≥ 1 ms between the two charges (parallel full-suite load, sandbox CPU throttling) admits the second charge,charge_inbound_framereturnstrue, and the assert fires. Every lib step (refill, refusal, retry clamp to [1 s, 60 s]) behaves as designed throughout.Fix (test-only,
pkgs/id/src/world_net.rs)expects, so every await in the test is bounded — a wedged charge now fails loudly in seconds instead of stalling a suite (which matters in the nix check phase, where the laptop setsmax-silent-time = 0and nothing reaps a silent build).Verification
world_net::tests22/22,abuse::tests12/12,frames::tests5/5. Edited region rustfmt-clean.cargo test -p id --libruns on the host are currently blocked by an unrelated, separately filed bug — every test that builds the clap command tree stack-overflows and aborts the binary on this host (id(cli): module_trust_keys test stack-overflows on host runs (unbounded recursion), aborts full-lib suite #5216; the same tests pass in the sandbox). The named-test and family verifications above stand in for it. Anix build .#id-webcheck-phase build of the fixed tree was started from the fix worktree and is in flight at closing time; the identical test target is compile- and run-verified on the host.Hang verdict (the 52-minute switch incident)
Not this test, on the evidence: 450 pre-fix runs with no hang, and a code audit showing every wait in its path is a tokio sleep of a refusal delay clamped to at most 60 s, with its substantive charges wrapped in 500 ms / 3 s timeouts (and now its first charges bounded too). A test binary sitting single-threaded at 0 CPU for 52 minutes is far more consistent with an unbounded network await elsewhere in a check-phase suite (the id lib binary carries live-iroh tests whose connects/receives have no outer timeout) than with this bounded test. The switch hang remains formally unattributed — it is not evidence against this test, and this fix does not claim to have repaired it.