run_process ignores thread context and spawns from threadpool #3360

Activity

1. A5rocks commented on Nov 26, 2025

   

   A5rocks

   on Nov 26, 2025

   Last edited by A5rocks

   Contributor

   More actions

   Looking at the docs, it sounds like we would want to use clone(2) to get a thread for sync stuff?

   Not all of the attributes that can be shared when a new thread is created using clone(2) can be changed using setns().

   Probably also the thread pool has some capacity limiter (I forgot!) so any change would have to beware... but maybe we can just call clone(2) if it's available to make a thread? I haven't looked at its manpage.

   (Disclaimer: I don't know my way around the syscalls available)
2. tik-stbuehler commented on Nov 27, 2025

   

   tik-stbuehler

   on Nov 27, 2025

   Author

   More actions

   Looking at the docs, it sounds like we would want to use clone(2) to get a thread for sync stuff?

   I strongly recommend against (even temporarily) cloning a new thread for every subprocess, and that would be the only way to make sure the subprocess inherits the current context.

   Actually, in my case a worker thread(pool) per "normal thread"/"trio loop" would do, because changing the namespace is the first thing I do, i.e. before calling trio.run; but I have multiple threads doing so with different namespaces. See code example at the bottom.

   Not all of the attributes that can be shared when a new thread is created using clone(2) can be changed using setns().

   No; I think this just means "not every CLONE_* flag is a namespace flag" - i.e. not every CLONE_* flag can be used with setns() (but I think all CLONE_NEW* flags actually can be used).

   Probably also the thread pool has some capacity limiter (I forgot!) so any change would have to beware... but maybe we can just call clone(2) if it's available to make a thread? I haven't looked at its manpage.

   Any high-level capacity limiter of a thread pool is unrelated to clone(2) and potential OS resource limits. But if you hit such an OS limit creating a new process is very likely to fail too.

   ---

   Attaching possible workaround for using a (thread-)local threadpool where needed (call use_local_thread_pool at least once in such threads):

   from __future__ import annotations
   
   from trio._core._thread_cache import ThreadCache, THREAD_CACHE as ORIG_CACHE
   import trio._core._thread_cache
   import threading
   import typing
   
   if typing.TYPE_CHECKING:
       import outcome
   
   
   _THREAD_LOCALS = threading.local()
   
   
   def use_local_thread_pool() -> None:
       # activate workaround for https://github.com/python-trio/trio/issues/3360
       if not hasattr(_THREAD_LOCALS, "pool"):
           _THREAD_LOCALS.pool = ThreadCache()
   
   
   class _StubCache:
       _idle_workers = ORIG_CACHE._idle_workers
   
       def start_thread_soon(
           self,
           fn: typing.Callable[[], trio._core._thread_cache.RetT],
           deliver: typing.Callable[[outcome.Outcome[trio._core._thread_cache.RetT]], object],
           name: str | None = None,
       ) -> None:
           if hasattr(_THREAD_LOCALS, "pool"):
               _THREAD_LOCALS.pool.start_thread_soon(fn, deliver, name)
           else:
               ORIG_CACHE.start_thread_soon(fn, deliver, name)
   
   
   trio._core._thread_cache.THREAD_CACHE = _StubCache()  # type: ignore

   ❤️React with ❤️1Reacted by A5rocks