to_thread is slow #4293

Activity

1. ngoldbaum commented on Aug 26, 2026

   

   ngoldbaum

   on Aug 26, 2026

   More actions

   I'm interested in this, however I'm not terribly familiar with zarr. I do see quite a bit of to_thread uses in the zarr implementation.

   Is there any chance you could put together a runnable example that demonstrates the slowness? That way I'll have a concrete example to share with the asyncio experts.
2. maxrjones commented on Oct 17, 2026

   

   maxrjones

   on Oct 17, 2026

   Member

   More actions

   here's a concrete example that demonstrates the slowness:

   # /// script
   # requires-python = ">=3.12"
   # dependencies = ["zarr", "numpy"]
   # ///
   """Zarr v2 arrays with compressor=None: chunk writes ~65% slower since zarr 3.3.0.
   
       uv run --with 'zarr==3.2.1' mvce_v2_uncompressed_write.py
       uv run --with 'zarr==3.4.1' mvce_v2_uncompressed_write.py
   
   Cause: since #3885, V2Codec wraps every chunk encode in asyncio.to_thread even when
   there is no compressor or filter, so the thread hop dominates a no-op encode.
   """
   import time, warnings
   import numpy as np, zarr
   from zarr.storage import MemoryStore
   
   warnings.simplefilter("ignore")
   data = np.arange(1_000_000, dtype="i4")  # 1000 chunks of 1000 int32
   
   def write(zarr_format, compressors):
       a = zarr.create_array(MemoryStore(), shape=data.shape, chunks=(1000,), dtype="i4",
                             zarr_format=zarr_format, compressors=compressors)
       a[:] = data
   
   def timeit(fn):
       t0 = time.perf_counter(); fn(); return (time.perf_counter() - t0) * 1000
   
   print(f"zarr {zarr.__version__}")
   for label, zf, comp in [("v2 compressors=None", 2, None), ("v2 default compressor", 2, "auto"), ("v3 compressors=None", 3, None)]:
       print(f"{label:24s} {min(timeit(lambda: write(zf, comp)) for _ in range(7)):7.1f} ms")

   ❤️React with ❤️1Reacted by Davis Bennett
3. Duo-ee commented on Oct 28, 2026

   

   Duo-ee

   on Oct 28, 2026

   More actions

   I reproduced the MVCE on Windows 11 (Python 3.12.14, zarr 3.4.1) and then measured the dispatch on its own, because the guess in the issue, "something faster than to_thread, e.g. a plain thread pool", turns out not to hold on this machine.

   This box was not quiet while I measured (other builds running on it), so absolute numbers move around between runs. I interleaved the variants round by round and report percentiles, so this is not a comparison of two different machine states.

   Dispatch cost on its own, no payload

   3000 interleaved rounds per variant after 300 warm-up rounds, no-op body:

   how the call is dispatched	min	p10	median	mean	p90
   direct call, no dispatch	0.7	1.0	1.1	1.2	1.3
   asyncio.to_thread	93.6	117.7	139.5	140.9	165.0
   loop.run_in_executor on a dedicated ThreadPoolExecutor(max_workers=8)	88.8	111.7	135.3	137.3	163.5
   loop.run_in_executor(None, ...)	95.2	116.2	138.4	140.4	165.9

   Microseconds per call. The three threaded variants are within 4 us of each other at every percentile. A hand-rolled pool does not buy anything here: the cost is the round trip to another thread and back, not the way asyncio.to_thread is spelled.

   The same thing on the real workload

   The MVCE array (1000 chunks of (1000,) i4), compressed and uncompressed, with np.array_equal checked on every variant so that a faster run cannot silently write different data. I ran the whole thing three times:

   case	to_thread vs direct sync call, write	to_thread vs direct sync call, read
   v2, compressors=None	1.78x / 2.08x / 2.07x	2.21x / 2.57x / 2.81x
   v2, default compressor	1.99x / 1.89x / 2.40x	2.13x / 2.41x / 2.45x

   The three numbers are three separate runs, each of them a min over 7 repeats.

   Measured inside the write path, one uncompressed 1000-element chunk takes about 6 us of actual _encode_sync work (median over 1000 chunks), while any threaded variant pays the dispatch on top of that. That ratio is the whole problem: the payload is an astype that in practice does not copy plus a bytes() view, two orders of magnitude smaller than a thread hop.

   It is not only V2Codec

   Per-chunk asyncio.to_thread is the pattern across the codec layer on main (a2f2415): src/zarr/codecs/blosc.py:332 and :350, gzip.py:68 and :82, zstd.py:85 and :99, numcodecs/_codecs.py:147, :150, :170, :173, :193, :196, and _v2.py:130, :137. v3 BytesCodec (bytes.py:133, :159) does its work inline, which is why v3 without a compressor did not regress.

   Caveat

   One Windows machine, and I could not run a Linux comparison (no WSL here). Neither the MVCE above nor this comment establishes what a thread hop costs on Linux; if it is an order of magnitude cheaper there, this is a Windows-shaped problem and that changes how much restructuring is worth it. Re-running the dispatch benchmark on Linux before deciding seems worth doing.

   Assuming the number holds up, there are two directions, and I would rather ask which one is acceptable than guess:

   1. Skip the hop when there is nothing to move off the loop: when filters is None and compressor is None, call _encode_sync inline. There is no blocking work in that branch, just the astype and the byte view.
   2. Keep one hop but amortise it: submit a batch of chunks per executor call so that 1000 chunks pay one hop instead of 1000.

   The second is the more general fix, but it changes how the pipeline schedules work, so it is presumably not a small PR.