Repository navigation
Autoscaler scales app to less than activation-scale #14017
Description
What version of Knative?
1.10.0
Expected Behavior
Services with activation-scale of "2" and initial-scale of "2" should always have at least two replicas running when not scaled to zero. e.g. when deployed the service will initially go to two replicas then if it receives no traffic for the scale to zero period it will go from 2 replicas to zero.
I'm not sure if this is strictly a bug or my misunderstanding of the expected outcome based on this description:
serving/pkg/apis/autoscaling/register.go
Lines 220 to 225 in bb043fa
ActivationScale is the minimum, non-zero value that a service should scale to.
Actual Behavior
Initially the service will correctly scale up to 2 replicas but if it does not receive traffic for the scale down period it will scale down to one replica before finally scaling to zero replicas.
If the service receives traffic (even one request) when one replica is running it will scale back up to two replicas.
Steps to Reproduce the Problem
Create a service with the following annotations:
autoscaling.knative.dev/activation-scale: "2"
autoscaling.knative.dev/initial-scale: "2"
Allow the service to scale down (by receiving no traffic)
Activity
dprotaso commentedon May 26, 2023
/triage accepted
What I observed:
When creating the new Knative service
When sending traffic
cc @psschwei in case you have input
xtreme-vikram-yadav commentedon May 26, 2023
/assign @xtreme-vikram-yadav
skonto commentedon May 29, 2023
+1 I have seen this too when trying that feature. I could not reproduce the expected behavior (but was not sure if I was missing anything).
psschwei commentedon May 30, 2023
Does the scale down from 2 --> 1 --> 0 happen every time, or just the first time? If it's just the first time, then the issue might be with initial scale.
From the release notes: Note that the initial target scale for a revision is still handled by
initial-scale;activation-scalewill only apply on subsequent scales from zero.(I think that got written after the code merged, but probably not a bad idea to update the comment to include that info)
It could also be that activation-scale only applies when scaling up... it's been a while and most of the discussion of this was on the old slack instance. That seems to sound right, based on what I remember, but again, it's been a while...
diarmuidie commentedon May 31, 2023
Confirmed that it happens every time (when the revision isn't getting traffic), not just the first time.
In my test case I have both
initial-scaleandactivation-scaleare set to 2.psschwei commentedon May 31, 2023
Looking at the docs for activation-scale (link), it seems that the intention was to cover just the scale up case (in other words, an initial scale that would be applied when scaling from zero).
For initial scale, we have the following in the docs: " After the Revision has reached this scale one time, this value is ignored. This means that the Revision will scale down after the initial target scale is reached if the actual traffic received only needs a smaller scale." It might make sense to add that to the activation scale section as well.
All that said, I'm in no way opposed to changing activation scale to expected behavior raised in this issue if that's what folks would prefer.
xtreme-vikram-yadav commentedon Jun 20, 2023
I think documenting the current behaviour of
activation-scaleshould suffice. Leaving the scale down decision to autoscaler makes sense unless there is a specific use case for support scaling down to zero by the value ofactivation-scale.@diarmuidie are you currrently affected by this or would improved documentation suffice?
Document current behaviour of scaling down with activation scale
Document current behaviour of scaling down with activation scale (#5619)
dprotaso commentedon Nov 21, 2023
@diarmuidie following up are you affected by the current behaviour?
We've done the documentation but it does not mean we shouldn't change it to be more 'intuitive' obvious.
diarmuidie commentedon Apr 22, 2024
Sorry about the delay, I somehow missed your comments. LGTM. Thanks for the doc update @dprotaso and @xtreme-vikram-yadav 👍