Repository navigation
Gracefully shutting down nurseries on KeyboardInterrupt #143
Description
I am experimenting with nested nurseries to handle multiple tasks associated with each incoming connection. I would like to send a final message on each connection as the nursery exits. I've run some tests and noticed a few things that seem to make this difficult.
- KeyboardInterrupt is only raised on the outer Nursery (at which point the nested nurseries, tasks, and incoming sockets have been dismantled)
KeyboardInterrupttriggers atrio.Cancelledin tasks (which can be caught) but not in children nursery scopes. If we push enough state down into the tasks this might be workable, but it would be easier to handle at the parent/nursery level.finallyseems the only way to trigger code as we leave a child nursery but as soon as we await, we exit (presumably because the scope has been cancelled)
What is the best way to trigger "awaitable" code (eg send one last message) as we exit a child nursery from a KeyboardInterrupt?
The following code illustrates the issue (without using any networking :-). I would like to trigger my shutdown code in the "inner" scope:
import trio
async def print_message(msg):
try:
while True:
print(msg)
await trio.sleep(1)
except trio.Cancelled:
print("Cancelled Print Message")
await trio.sleep(1)
print("Cancelled Blocking Print Message")
finally:
print("Finally Print Message")
await trio.sleep(1)
print("Finally Blocking Print Message")
async def print_spawner(nursery):
count = 0
while True:
nursery.spawn(print_message, "msg {}".format(count))
count += 1
await trio.sleep(1)
async def child():
try:
async with trio.open_nursery() as inner_nursery:
inner_nursery.spawn(print_spawner, inner_nursery)
except KeyboardInterrupt:
print("Inner Keyboard")
except trio.Cancelled:
print("Inner Cancelled")
await trio.sleep(1)
print("Inner Blocking Cancelled")
finally:
print("Finally Inner")
await trio.sleep(1)
print("Finally Blocking Inner")
print("Inner Nursery Finished")
async def parent():
try:
async with trio.open_nursery() as outer_nursery:
outer_nursery.spawn(child)
except KeyboardInterrupt:
print("Outer Keyboard")
except trio.Cancelled:
print("Outer Cancelled")
await trio.sleep(1)
print("Outer Blocking Cancelled")
finally:
print("Finally Outer")
await trio.sleep(1)
print("Finally Blocking Outer")
print("Outer Nursery Finished")
trio.run(parent)
msg 0
msg 0
msg 1
^CCancelled Print Message
Cancelled Print Message
Finally Print Message
Finally Print Message
Finally Inner
Outer Keyboard
Finally Outer
Finally Blocking Outer
Outer Nursery Finished
Activity
njsmith commentedon May 1, 2017
This is partly a matter of luck –
KeyboardInterruptcan get raised absolutely anywhere. But it does tend to end up in the outermost task because that's who gets it if theKeyboardInterruptarrives at a moment when no tasks are running. You can think of this as, it's getting raised right at the end of the outermost nursery block, and then triggering the regular nursery cleanup logic. Which, of course, starts by cancelling that outermost nursery's cancel scope.What do you mean that children nursery scopes don't get a Cancelled exception? Once the outermost nursery has its cancel scope cancelled, then everything (children, grandchildren, great-grandchildren, ...) below it should get a
Cancelledexception the next time it executes a checkpoint.Oh, actually, I have a guess – there are several grandchildren that get Cancelled exceptions, and those propagate into their parent, one of the first-level children. And since it's supervising multiple grandchildren and they all crashed, it bundles their exceptions up into a
MultiError. So if you doexcept Cancelled:, that won't catch aMultiErrorrepresenting multipleCancelleds. A simple check would be to catchBaseException as excandprint(repr(exc)). UnfortunatelyMultiErrorsare not super easy to work with because they don't fit very neatly into Python's idea of how exceptions work, but there are some tools to help, in particularwith MultiError.catch(filter_callback): ..., wherefilter_callbackgets called with all the individual exceptions and can let them continue to propagate, catch them, or replace them with a different exception.As noted above, catching
MultiErrorwill also work, but if you want to run something every time you exit thenfinallyis a better choice, or if you want to run something on all exceptional exits then anexcept: ... raiseblock is totally legit. But yes, once a scope has been cancelled, all blocking operations are disabled. See this section, in particular the discussion of "level triggered" cancellation.So the fundamental challenge here is that there isn't really any way to know "why" you were cancelled – maybe the peer has totally disappear and a timeout finally expired, maybe there was a KeyboardInterrupt and the program is shutting down, ... and this is intentional, because it turns out that once you allow multiple "types" of cancellation it makes life really messy really fast. What if a cancel scope had its timeout expire and someone called
.cancel()on it explicitly, how should that be signaled, that kind of thing.One option is discussed in that doc section I linked above: while a scope that's been cancelled prevents all blocking operations inside it by default, you can disable this by wrapping your "say goodbye" code in a new cancel scope with its
shieldattribute set toTrue. I strongly recommend that if you do this, you also set a sensible timeout on the shield cancel scope, because if the remote host doesn't respond or whatever then your goodbye code will get forever and there's nothing anything outside the shield can do to stop it. But for your use case a timeout of like 0.1 seconds might be plenty, if it's just a best effort attempt to stick a few bytes on the wire before exiting. Of course, this will then run no matter why your code was cancelled.Alternatively, if you really want control-C specifically to be treated differently than other kinds of shutdown/crash/cancellation events, you absolutely can set up a custom handler for it that does whatever clever thing you want. For example, you could set a global flag
exiting_due_to_KeyboardInterrupt = Trueand then raise KeyboardInterrupt, and then your individual tasks could consult that flag. There's an example here. The downside is that this kind of custom handler doesn't work if your code gets stuck in an infinite loop and isn't letting trio schedule tasks, but life always has trade-offs... I guess it would also be possible to make a custom signal handler that sets that flag and then invokes trio's normal control-C logic. There's some discussion of this in #134, but if you want to hack it you could do something like:Does any of that help? It's kind of hard to give specific advice without knowing what you're ultimately trying to do :-)
merrellb commentedon May 1, 2017
Thanks for the detailed response. My ultimate use case isn't all that mysterious (#124 :-) Basically I want to gracefully send the close message to each WebSocket connection when shutting down the server.
As an async/trio newbie, I am still struggling to wrap my head around the idea that the child nurseries aren't nested in the typical stack sense, ready to catch any exception before a parent nursery could possibly see it (eg if all of my leaf nurseries wrap
KeyboardInterruptat least one of them should catch it). However, it does make sense that while idle it wouldn't be clear where, other than the outer nursery, to raise the Exception (and regardless it seems a bad idea to depend on the vagaries of what happens to be executing when one hits Control-C)It was the
MultiErrorthat was keeping me from catching the nurseryCancelledalthough I thinkfinallyis what I really need to gracefully close connections/nurseries.In
finally,open_cancel_scopewithshield=Trueseems to do the trick. However, given the importance of timeouts, I am a bit surprised to see none of the convenience functions (egmove_on_after) accept theshieldargument (feature request? :-)As always, thanks for your patience as I try to figure this stuff out.
merrellb commentedon May 2, 2017
I've created a new issue for adding
shieldto the convenience functions (#147) and will close this issue as there doesn't seem to be much else to add.