Repository navigation
Is handling SIGTERM in user containers counterproductive? #13600
Description
- user-container which is a golang HTTP server.
- handles SIGTERM and performs
Shutdown- starts by terminating the listener, preventing new connections
Looking at how queue-proxy works, is this actually counterproductive / going to lead to additional issues?
The specific scenario I am imagining is as follows:
┌────────────────────────┬────────────────────────┐
│queue-proxy │user-container │
│ │ │
Connection │ │ │
────────────-───┼─┐ Successfully reaches │ │
│ │ proxy │ │
────┼─┼─────────────────SIGTERM──────────┬────────────┼──┬───
│ │ │ │ │ │
│ │ │ ┌──────┴────────┐ │ │
│ │ Drain │ │ Listener │ │ │
│ │ │ │ shutdown │ │ │
│ │ │ └──────┬────────┘ │ │
│ │ │ │ │ │
│ └──────────────────────┴─────────►(Fails) │ │
│ Request attempted │ │ Grace period
│ │ │ │
│ │ │ │
│ │ │ │
│ │ │ │
│ │ │ │
│ │ │ │
│ │ │ │
└────────────────────────┴────────────────────────┘ ▼
That is:
a) An incoming connection reaches the queue proxy
b) SIGTERM is triggered before queue proxy makes a request to the user container
c) The user container shuts down its HTTP listener
d) queue-proxy attempts to forward the request to user container, and now fails because the user container has stopped listening.
Is this a realistic scenario that could occur, or are there reasons that this is not actually possible in reality?
And in general, for a stateless HTTP server, is there value in handling SIGTERM when working with knative serving - is it actually recommended?
Activity
mastersingh24 commentedon Feb 9, 2023
Might want to have a read through #2311. It's an interesting read.
dprotaso commentedon Mar 17, 2023
The user container should shutdown gracefully when it receives a SIGTERM.
This isn't accurate. Knative uses a PreStop to inform that the user container is going to shutdown. We hold this hook open so that we can drain connections. Only after we drain do we let the hook return and the user-container receives the SIGTERM.
If you don't handle the SIGTERM and shutdown then the Pod will continue running until K8s sends a SIGKILL. This happens after
terminationGracePeriodwhich we usually set to the Revision's Timeout. We do this in case a request comes in while shutting down and want to give it enough time to complete.wenzhaojie commentedon Jun 20, 2023
@dprotaso I'm currently facing an issue where the SIGTERM signal doesn't seem to be promptly delivered to the user-container. My user-container is a simple HTTP server that can correctly capture the SIGTERM signal and exit the container. However, when the pod shows a Terminating state, my user-container doesn't receive this signal immediately (with no remaining requests), but rather receives the SIGTERM after 30 seconds, and then proceeds to shut down and exit.
Is the user-container immediately notified when the Pod enters the Terminating state? Thanks a lot!
dprotaso commentedon Jun 20, 2023
@wenzhaojie That's by design. It takes some time for the networking layer to remove your Pod's endpoint. So we purposely keep the Pod alive for at least ~30-45 seconds and exit if there are no active requests.
ashrafguitoni commentedon Feb 27, 2024
Quick tip for people whose user containers aren't terminating gracefully, try using
dumb-initor similar solutions.