KeyError in KqueueIOManager.process_events (TrioInternalError)

We have a PySide6 (Qt) application using Trio in guest mode. We are occasionally seeing the error below:

Traceback (most recent call last):
  File "/Users/dev/spire/.venv/lib/python3.14/site-packages/trio/_core/_run.py", line 2765, in unrolled_run
    runner.io_manager.process_events(events)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
  File "/Users/dev/spire/.venv/lib/python3.14/site-packages/trio/_core/_io_kqueue.py", line 93, in process_events
    receiver = self._registered[key]
               ~~~~~~~~~~~~~~~~^^^^^
KeyError: (46, -2)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/dev/spire/spire-desktop/src/quicore/asyncio.py", line 142, in event
    event.fn()
    ~~~~~~~~^^
  File "/Users/dev/spire/.venv/lib/python3.14/site-packages/trio/_core/_run.py", line 1752, in guest_tick
    self.done_callback(Error(exc))
    ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
  File "/Users/dev/spire/spire-desktop/src/quicore/asyncio.py", line 130, in _done_callback
    result = outcome_.unwrap()
  File "/Users/dev/spire/.venv/lib/python3.14/site-packages/outcome/_impl.py", line 213, in unwrap
    raise captured_error
  File "/Users/dev/spire/.venv/lib/python3.14/site-packages/trio/_core/_run.py", line 1746, in guest_tick
    timeout = self.unrolled_run_next_send.send(self.unrolled_run_gen)
  File "/Users/dev/spire/.venv/lib/python3.14/site-packages/outcome/_impl.py", line 185, in send
    return gen.send(self.value)
           ~~~~~~~~^^^^^^^^^^^^
  File "/Users/dev/spire/.venv/lib/python3.14/site-packages/trio/_core/_run.py", line 2945, in unrolled_run
    raise TrioInternalError("internal error in Trio - please file a bug!") from exc
trio.TrioInternalError: internal error in Trio - please file a bug!

Unfortunately, I'm not able to reproduce the error reliably, nor am I certain about where to go looking for the issue. Any insight would be most appreciated.

Edit: nvm my previous suggestion probably wouldn't help... hmm, maybe you can poke at Trio internals to see what gets registered for kqueue?

@james-emerton

@james-emerton could you check the hypothesis in #3502? Probably this involves:

adding logging around any notify closings
decorate the function passed into run_sync_soon to log before/after it finishes.
log when run_sync_soon gets called

I believe this should look something like this, according to that hypothesis:

after run_sync_soon's function
called run_sync_soon
called notify_closing with 42
before run_sync_soon's function
Key error: (42, ...) not in self._registered

Also I assume you're running on non-free-threaded Python?

(Alternatively I guess you could run the code from that branch and see if you still get these errors, though I want to make sure we're being more lenient for the right reason instead of covering up a deeper flaw)

@A5rocks

Hello @A5rocks

Thanks very much to both you and @afonsojanu for looking at this.

Unfortunately, we are experiencing this error extremely infrequently. I have two documented occurrences and no more data since I made the report here. I have attempted to reproduce the error as reported to me (opening a window and closing it right away which should start and then cancel a coroutine.) We have a large non-async codebase that's just starting to migrate a few things over; and most users at this time are Windows users so there just isn't that much opportunity for us to step on this yet.

If I understand the issue correctly, it's that in guest mode it is possible to have a wait signal but have the coroutine cancelled before the signal event is processed. Perhaps it would be possible to reproduce it more deliberately by spawning and cancelling large numbers of tasks while running in guest mode?

Edit to add that we're on Python 3.14.2 with the GIL

I believe you can cause it to happen more by having the run_sync_soon you pass take longer before actually executing the function trio passes. But idk

Hmm, do you use trio.lowlevel.current_kqueue anywhere? The only two ways so far I can think to get this error are:

using our exposed kqueue without adding a relevant listener, which is a bug in your usage though we could improve error messages lol (would need trio.lowlevel.current_kqueue and I think need to call .control, as things to search for)
using trio.lowlevel.monitor_kevent, which is a bug I have an open PR to fix. (just that should be enough to search for)

Hopefully you can audit your code for either of those!

Generally it feels like patching this by adding a .get like the LLM PR did is simply a band-aid and masking other bugs -- if you get an event, you should know who's waiting for it!!

I asked an LLM about this and it managed to reproduce the issue with a deterministic test with a synthetic host loop. In short, the issue happens if an IO event completes (so it is in the results returned from get_events()) but then a non-Trio function running in the host event loop calls a Trio cancellation that cancels that IO before the successful result is handled (process_events()) in the Trio event loop.

There are two lists that need to be kept matching: io_manager._kqueue which is the OS objects in the kqueue itself, and io_manager._registered which is just a regular Python dictionary (mapping from event key to the Trio task that should be woken by it).
Both the successful return and the cancellation fight over deleting from these two lists:

get_event() completes with success for this key so the OS automatically removes it from io_manager._kqueue (because it had KQ_EV_ONESHOT in the flags).
When the host event loop calls cancel() it synchronously runs these steps:
- It calls EV_DELETE to remove the event from io_manager._kqueue, which fails with ENOENT (it's already removed by step 1) but this is silently swallowed.
- It deletes io_manager._registered[event_key]
- It reschedules the task with Cancelled.
When the Trio event loop is resumed in the host thread, and gets to process_events(), it wants to lookup the task with io_manager._registered[event_key] and then delete that key and reschedule the task - but the key is already gone, so here we get the crash. (And, the task was already rescheduled.)

The LLM's suggested fix is actually exactly #3502: process_events() should ignore missing keys, assuming that the above sequence is what just happened. It's the right fix but, just with the wrong justification.

[Edit: I was going to write a much longer write up about this but I just expanded / fixed my explanation in this post and this short version is probably more useful than the essay I was about to write.]

-- Expand this collapsable section to see a self-contained reproducer on BSD/MacOS (written by the LLM) --

# Reproducer for https://github.com/python-trio/trio/issues/3500
# Run on macOS or a BSD (kqueue backend). Expected on an unfixed Trio:
#   trio.TrioInternalError ... caused by KeyError: (<fd>, -2)
import queue
import socket

import outcome
import trio
import trio.lowlevel


def main():
    todo = queue.Queue()  # the "host loop": a FIFO of callbacks, run in order

    async def trio_main():
        a, b = socket.socketpair()
        with a, b, trio.CancelScope() as cancel_scope:
            # Queued on the host loop now, so it runs after this tick has
            # fetched the "writable" event for `a`, but before the tick that
            # delivers it.
            todo.put(cancel_scope.cancel)
            await trio.lowlevel.wait_writable(a)  # `a` is writable already
        assert cancel_scope.cancelled_caught

    trio.lowlevel.start_guest_run(
        trio_main,
        run_sync_soon_threadsafe=todo.put,
        run_sync_soon_not_threadsafe=todo.put,
        done_callback=todo.put,
    )
    while True:
        item = todo.get()
        if isinstance(item, outcome.Outcome):
            item.unwrap()
            print("ok: cancelled cleanly")
            return
        item()


if __name__ == "__main__":
    main()

Generally it feels like patching this by adding a .get like the LLM PR did is simply a band-aid and masking other bugs -- if you get an event, you should know who's waiting for it!!

This is fair and I think it's possible in principle to avoid but a lot of work. The outline is:

If a host callback in guest mode calls cancel() (or change shield to False in some situations), then set cancel_called but we just note that cancellation but don't cancel or reschedule anything (with recalculate()), and wake the Trio loop (with force_guest_tick_asap()).
when the Trio event loop resumes, io_manager.process_events(events) runs next, as usual
then (before deadlines.expire(clock.current_time())) we go through the cancellations we noted earlier and make them take effect now (call recalculate()).

In the race we've been discussing, the IO wins and the cancellation of the scope does nothing (because it can see the task is already rescheduled due to process_events()).

The LLM says that you'd need to do the same thing from notify_closing and monitor_kevent as they have the same issue.

It also means that abort functions are not run synchronously from cancel() in this situation, which would be a user visible change in behaviour. I can't think of an obvious consequence but it seems risky to me.

The LLM also found two very similar bugs (which I've attempted to summarise in Humanese):

notify_closing from host loop races with success: This is the bug discussed in #3503. It would be the same as the cancellation race, but with notify_closing() instead of cancel() (get_events() finishing then notify_closing() deleting from io_manager._registered[event_key] before process_events() reads it)... except, additionally, notify_closing() doesn't silently absorb ENOENT/EBADF so OSError escapes to the caller (and the KeyError in step 3 doesn't happen after all). There are two possible fixes here:

The one in #3503, which detects the ENOENT/EBADF and stops any other processing for this task in step 2 - in other words, the task gets woken by step 3 as originally planned. This is a totally self-contained fix for this problem. It does have the downside that the task will be told the FD is readable/writable rather than ClosedResourceError. But that is actually possible in Trio even not in guest mode, if you close a socket when a reader/writer has already been scheduled to resume.
notify_closing(), when EV_DELETE gets ENOENT/EBADF, just carry on (del io_manager._registered[event_key], reschedule(task, outcome.Error(ClosedResourceError(...)))). This requires the other bug to be fixed already with the fix from #3502 (because, in step 3, io_manager._registered[event_key] is now missing). This does inject ClosedResourceError into the waiting task.

Process.wait() cancellation from host loop races with success: This is similar to the notify_closing() bug, but it doesn't go through exactly the same cancellation path (_subprocess_platform/kqueue.py uses its own abort function rather than going through io_manager._wait_common()) so it also doesn't check for ENOENT. That means an exception escapes from the abort function which propagates up through cancel(), exiting it at a random time in an inconsistent state. The fix here is to catch ENOENT in the abort function, and also depends on the fix in #3502.

Test cases (one for each of the three bugs) and fixes can be found in arthur-tacca@52e8d14. I'm afraid this is the raw unedited output: I think the fixes and the tests are useful but of course it has LLMisms in its comments (and it wrote a newsfragment - yuck!)

Here are my rambling event loop notes, probably no use to anyone except me. (Steps 1, 2, 3 in my comments above map to step 2 in worker thread, step 2 in host thread, and step 3, respectively.) https://gist.github.com/arthur-tacca/9f73fc8054cf7a816b210c58f4f1a499

@A5rocks

Quoting @A5rocks:

Hmm, do you use trio.lowlevel.current_kqueue anywhere?

No, we use very little from trio.lowlevel; only start_guest_run and in_trio_task.

To start an async task from a sync context, we use a channel to pass task functions into the main loop, where it spawns them into a nursery. Typically, tasks are bound to the lifespan of a QObject (usually a window), so that when a window is closed the associated tasks are cancelled.

Generally it feels like patching this by adding a .get like the LLM PR did is simply a band-aid and masking other bugs -- if you get an event, you should know who's waiting for it!!

This is fair and I think it's possible in principle to avoid but a lot of work. The outline is:

If a host callback in guest mode calls cancel() (or change shield to False in some situations), then set cancel_called but we just note that cancellation but don't cancel or reschedule anything (with recalculate()), and wake the Trio loop (with force_guest_tick_asap()).

when the Trio event loop resumes, io_manager.process_events(events) runs next, as usual

then (before deadlines.expire(clock.current_time())) we go through the cancellations we noted earlier and make them take effect now (call recalculate()).

In the race we've been discussing, the IO wins and the cancellation of the scope does nothing (because it can see the task is already rescheduled due to process_events()).

The LLM says that you'd need to do the same thing from notify_closing and monitor_kevent as they have the same issue.

It also means that abort functions are not run synchronously from cancel() in this situation, which would be a user visible change in behaviour. I can't think of an obvious consequence but it seems risky to me.

Yeah this is convincing to me. So I guess in summary, 4 bugs:

getting events, doing something, processing them:
- if that something is notify_closing, then deleting kqueue events can raise even though there's a registered thing (solved by Sometimes notify_closing can be called on dead events #3503 )
- if that something is .cancel(), then the abort gets called, which removes the registered thing before process events happens, causing a keyerror (TODO: .get the key. maybe we could try to detect invalid events by keeping some extra state but that sounds hard, similar to Ignore stale kqueue events for an already-deregistered key #3502)
- there is probably a bug w/r/t processes, though I haven't wrapped my head around it (would be fixed by KeyError in KqueueIOManager.process_events (TrioInternalError) #3500 (comment) by making it one-shot)
not mentioned above but I found it elsewhere: wait_readable/wait_writable can add an infinite number of registrations (? or just reset things?) to the same kqueue, because we add to kqueue and only then check if we already registered something. (KeyError in KqueueIOManager.process_events (TrioInternalError) #3500 (comment))

I guess we have some work to do :^)

wait_readable/wait_writable can add an infinite number of registrations (? or just reset things?) to the same kqueue, because we add to kqueue and only then check if we already registered something

Good point. I don't think you can get multiple registrations, because the docs say "Re-adding an existing event will modify the parameters of the original event, and not result in a duplicate entry." I think even modifying the parameters is not a practical problem, as Trio uses consistent parameters for each filter type. But it would be sensible to do the check: we just need to duplicate if key in self._registered: raise _core.BusyResourceError(...) from these lines in wait_kevent() into _wait_common().

For a better fix: The root cause is that wait_kevent() checks whether the item is in _registered after registering in the OS kqueue itself. The current API forces that because the caller has already done the registration before the function is called. So the real fix would be to change wait_kevent() to do the registration itself (and handle cancellation), so it can do the _registered check first. I think this is actually the better API anyway: it makes _wait_common() redundant, and simplifies wait_child_exiting() a lot. It's technically less flexible (it would only handle one-shot events, and not events implicitly created by some other system call), but I think that's OK. It's also a breaking API change but I doubt there are any callers outside of Trio itself (there aren't according to GitHub search). It might look like this (comments omitted for brevity - and not tested!):

class KqueueIOManager:
    @_public
    async def wait_kevent(
        self,
        ident: int | _HasFileNo,
        filter: int,
        *,
        fflags: int = 0,
        data: Any = 0,
    ) -> select.kevent:
        if not isinstance(ident, int):
            ident = ident.fileno()

        key = (ident, filter)
        if key in self._registered:
            raise _core.BusyResourceError(
                "attempt to register multiple listeners for same ident/filter pair",
            )

        # Register the event before updating _registered in case kevent.control() throws
        flags = select.KQ_EV_ADD | select.KQ_EV_ONESHOT
        event = select.kevent(ident, filter, flags, fflags, data)
        self._kqueue.control([event], 0)

        self._registered[key] = _core.current_task()

        def abort(raise_cancel: RaiseCancelT) -> Abort:
            event = select.kevent(ident, filter, select.KQ_EV_DELETE)
            del self._registered[key]

            try:
                self._kqueue.control([event], 0)
            except OSError as exc:
                if exc.errno in (errno.EBADF, errno.ENOENT):  # pragma: no branch
                    pass
                else:  # pragma: no cover
                    # As far as we know, this branch can't happen.
                    raise

            return _core.Abort.SUCCEEDED

        return await _core.wait_task_rescheduled(abort)  # type: ignore[no-any-return]

    @_public
    async def wait_readable(self, fd: int | _HasFileNo) -> None:
        await self.wait_kevent(fd, select.KQ_FILTER_READ)

    @_public
    async def wait_writable(self, fd: int | HasFileNo) -> None:
        await self.wait_kevent(fd, select.K_Q_FILTER_WRITE)

async def wait_child_exiting(process: _subprocess.Process) -> None:
    try:
        from select import KQ_NOTE_EXIT
    except ImportError:  # pragma: no cover
        KQ_NOTE_EXIT = 0x80000000  # type: ignore[misc]
    try:
        await _core.wait_kevent(process.pid, select.KQ_FILTER_PROC, fflags=KQ_NOTE_EXIT)
    except ProcessLookupError:  # pragma: no cover
        pass

This would also fix the wait_child_exiting() bug in passing (when combined with #3502), because it gets the same suppression of spurious error.errno as _wait_common currently does. I also fixed the return type of wait_kevent() from Abort to select.kevent.

I'm happy to prepare a pull request, but I think the other two pull requests should be merged first to avoid sorting out a conflict later (#3502 with modified comment, and #3503).

Edit: or I could make a pull request that bundles #3502, #3503, and the above fix all at once, with the three LLM generated tests (which I think are decent).

Hm the issue is that we don't want to register everything as one shot, eg monitor_kqueue (which doesn't use wait_kevent tbh). I guess the abstraction makes sense though, cause wait_kevent without one shot implies a race condition? Eg if the event fires twice before trio gets events.

On further thought I think a new wait_kevent makes sense. I'd rather separate this from the .get thing -- you can test this similar to how #3503 did (before I force pushed that away), i.e. that monitor_kqueue and then wait_readable/wait_writable doesn't overwrite that.

I think we can preserve backwards compatibility by adding an extra kwarg for abort and, if it is provided:
a) putting up a deprecation warning
b) avoid adding anything to kqueue

I think that should be simple enough? And it shouldn't conflict! (also idk when someone will get around to reviewing #3503 so I wouldn't wait around...)

On further thought I think a new wait_kevent makes sense.

I think that's probably for the best, but I did reconsider after reading your comment and here are some rambling notes about how I came to that conclusion.

Coming up with a better API for custom kevent listening turns out to be incredibly complex. Partly because of the answer to this:

I guess the abstraction makes sense though, cause wait_kevent without one shot implies a race condition? Eg if the event fires twice before trio gets events.

There's a race condition if a caller uses a non-oneshot event and doesn't await wait_kevent() every event loop iteration (one of the results would be discarded) but that's their problem.

If you just mean the event fires twice in one Trio event loop iteration, then actually it's the opposite: for some filters, using wait_kevent (repeatedly in a loop) races only if it is used with a one shot. Without one shot, for some filters, if the event fires twice from one return of get_events() to the next then the OS updates information in the event in some way to let you know e.g. for EVFILT_SIGNAL there's a count of number of times a signal fired since the last report.

The other reason a general API is hard is that there are two cases where events get registered by a different method than a simple _kqueue.control([kevent(ident, filter, ...], ...): AIO/LIO (uses a separate API) and EVFILT_PROC with NOTE_TRACK which can add events under the pid of the child (or grandchild, etc.).

I've been on a whole journey trying to figure out a good API which fixes the registration order problem:

The simple version at the top of my previous comment: just add an extra check to _wait_common - but this doesn't help direct users of wait_kevent
The main suggestion from my previous comment: force wait_kevent to be oneshot, get the registration check right; but that excludes callers that aren't oneshot or another API
Add a context manager: it checks/adds to _registered on entry, leaves it to the caller to do the OS call to register the event, has an async method to wait for it to fire, then unregisters and EV_DELETE on exit. But this didn't work as it meant a cancellation didn't immediately remove the registration (both OS and _registered) until the waiting task resumed, so you can't cancel it and immediately wait the same event; this is a breaking change visible for regular wait_readable / wait_writable.
Pass callbacks to wait_kevent to register and unregister with the OS, slotted in at the correct time in the _registered checks / changes. But this delegates so much work to the caller that it seemed hardly better than just making a new check_registered() method which advanced callers can with existing wait_kevent.

After all that I went back and read all the old issues on the topic and was surprised to find that the most recent comment about it by NJS (which he's linked to a couple of times since) actually backs off from leaving the user to do the registration: AIO seems unlikely to be used from Python (but not impossible) and it's BSD only, and EVFILT_PROC with NOTE_TRACK doesn't work with the current API due to its unpredictable ident field. "So, never mind all that. Let's drop current_kqueue. " ... "I think we probably want to public functions: one that listens for a single ONESHOT event, and one that listens for events on an ongoing basis." It seems like the issue didn't progress just because of inactivity rather than because of any objection to this. So we're finally implementing that first part (and the second part, monitor_kevent, can wait for another day).

I think we can preserve backwards compatibility by adding an extra kwarg for abort and, if it is provided:
a) putting up a deprecation warning
b) avoid adding anything to kqueue

I cannot believe that there are any existing callers (outside of Trio). But, sure.

Fix python-trio#3500: TrioInternalError on macOS/BSD in guest mode

wait_kevent: do the kevent registration itself (part of python-trio#578)

Navigation Menu

Repository navigation

KeyError in KqueueIOManager.process_events (TrioInternalError) #3500

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

KeyError in KqueueIOManager.process_events (TrioInternalError) #3500

Description

Activity

A5rocks commented on Sep 1, 2026

A5rocks commented on Sep 4, 2026

james-emerton commented on Sep 4, 2026

A5rocks commented on Sep 4, 2026

A5rocks commented on Sep 8, 2026

arthur-tacca commented on Sep 8, 2026

arthur-tacca commented on Sep 8, 2026

arthur-tacca commented on Sep 8, 2026

arthur-tacca commented on Sep 8, 2026

james-emerton commented on Sep 8, 2026

A5rocks commented on Sep 9, 2026

arthur-tacca commented on Sep 16, 2026

A5rocks commented on Sep 16, 2026

A5rocks commented on Sep 17, 2026

arthur-tacca commented on Sep 17, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions