Repository navigation
pod.TerminationGracePeriodSeconds setting #15599
Description
Describe the feature
Right now TerminationGracePeriodSeconds is set to rev.Spec.TimeoutSeconds
https://github.com/knative/serving/blob/main/pkg/reconciler/revision/resources/deploy.go#L304
However, rev.Spec.TimeoutSeconds also specifies the timeout for in-flight request.
I think these two values should be seperated, because in my project, I want to terminate deployment without graceful exit, but I want the timout for in-flight request to be as long as possible.
Activity
skonto commentedon Oct 30, 2024
Hi @V2arK, this was added years ago so there is a guarantee about connections not to be dropped during autoscaling. The knative autoscaler continuously makes decisions about the deployment scale and that may interrupt connections during pod shutdown.
Could you elaborate on your use case, you don't care about failing requests?
V2arK commentedon Oct 31, 2024
Hi @skonto, in my uses cases I just want to terminates the pods ASAP (maybe 3~5 seconds) when I triggers the termination, but not to change the timeout for requests (eg, LLM spits out response in minutes),
skonto commentedon Nov 27, 2024
Hi @V2arK
How do you trigger that? You are removing the knative service? Could you just drop the connection when a SIGTERM is received at the LLM python runtime or at the client side too? If you interrupt the connections, draining will happen pretty fast as QP will not wait for them to finish.
V2arK commentedon Dec 17, 2024
Hi @skonto ,
Sorry for the late reply; your message got buried among other communications, and I missed it.
In our workflow, we use ArgoCD to deploy a
serving.knative.dev/v1Helm chart. When signaling a termination or deletion of a deployment via ArgoCD, we aim to delete the entire deployment quickly. Ideally, we would like Kubernetes to send aSIGKILLdirectly, (so we can save as much dime as we could :) ) which is controlled byrev.Spec.TimeoutSeconds.However, we don't want to change the timeout for in-flight requests; we want it to remain as long as possible.
github-actions commentedon Mar 18, 2025
This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with
/reopen. Mark the issue asfresh by adding the comment
/remove-lifecycle stale.dprotaso commentedon Apr 18, 2025
FYI - it's usually the user's app that isn't exiting properly (handling SIGTERM) that's keep the pod alive (and not Knative)
See: #13600
dprotaso commentedon Apr 18, 2025
This issue does seem related - #15823