Skip to content

bug(test): test_sync_timeout races a 2 ms timeout against a 20 ms sleep — failed twice on main (macos-latest, py=3.12) #4582

Open

Description

@glaziermag

tests/test_sync.py::test_sync_timeout has failed twice on main since 09-23. Both failures were on os=macos-latest, py=3.12, deps=optional, and both were Failed: DID NOT RAISE TimeoutError:

The test's coroutine sleeps for 20 ms, and the test expects sync(foo(), timeout=0.002) to time out (tests/test_sync.py#L63-L70). sync() submits the coroutine with run_coroutine_threadsafe and then calls concurrent.futures.wait(..., timeout=timeout) (src/zarr/core/sync.py#L147-L151). Suppose the test thread is not running again within about 18 ms of submitting. Then the coroutine has already finished, wait() returns it in finished, and sync() returns normally. On a busy runner, that 18 ms is not guaranteed.

Reproduction. This version is deterministic: it delays the test thread by 50 ms between submitting the coroutine and waiting on it.

import asyncio
import time

import zarr.core.sync as zsync

real_wait = zsync.wait


def late_wait(*args, **kwargs):
    time.sleep(0.05)  # stands in for the test thread being descheduled
    return real_wait(*args, **kwargs)


zsync.wait = late_wait


async def foo() -> None:
    await asyncio.sleep(0.02)


zsync.sync(foo(), timeout=0.002)
print("returned without TimeoutError")

Without the patch, the failure is rare. The unmodified test body was run on an 8-core Mac with 24 busy loops alongside it. It missed the timeout in 1 of 2,000 iterations in each of two runs, and in 0 of 4,000 in a third. With the machine idle, it missed in 0 of 1,000.

One way to remove the timing dependence is to keep the coroutine unfinished until the test has seen the timeout:

@@ -1,2 +1,3 @@
 import asyncio
+import threading
 from collections.abc import AsyncGenerator
@@ -63,9 +64,15 @@ def test_sync_raises(sync_loop: asyncio.AbstractEventLoop | None) -> None:
 def test_sync_timeout() -> None:
-    duration = 0.02
+    release = threading.Event()
 
     async def foo() -> None:
-        await asyncio.sleep(duration)
-
-    with pytest.raises(asyncio.TimeoutError):
-        sync(foo(), timeout=duration / 10)
+        # Stay unfinished until the timeout has been seen, so the result does not depend
+        # on how soon the calling thread is rescheduled after submitting the coroutine.
+        while not release.is_set():
+            await asyncio.sleep(0.001)
+
+    try:
+        with pytest.raises(asyncio.TimeoutError):
+            sync(foo(), timeout=0.002)
+    finally:
+        release.set()

With this change, the delayed-wait setup above still raises TimeoutError in 50 of 50 iterations. pytest tests/test_sync.py also passes (19 passed) under the project's pytest configuration.

The open PR #3997 adds test_run_timeout, which uses the same 20 ms sleep with timeout=duration / 10, so it would hit the same race. The future.cancel() added in #4343 does not change this, because it only runs when wait() reports the future as unfinished.

This was investigated and written up with Claude Code, an AI agent. Before filing, the two run logs were re-read, and the reproduction and the patched test were run on macOS arm64 with Python 3.12.12 and zarr at bf0d231.

Activity

  1. sheikhayaan commented on Oct 31, 2026

    @sheikhayaan
    Contributor

    I'd like to pick this up. Using a threading.Event to keep the coroutine unfinished until the timeout is seen seems like the most robust fix for the flakiness. I'll apply the patch to test_sync_timeout and open a PR. Should I also update test_run_timeout in PR #3997 here, or handle that separately?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions