Route promotes a Revision to 100% traffic while it is still below initial-scale (latestReadyRevision latches an under-scaled Revision)

/area autoscale
/area API

/kind spec

What version of Knative?

Knative Serving v1.20.0 (observed). Root-cause code is unchanged on main and in the latest
release knative-v1.22.1, so all released versions through v1.22.1 are affected.
Related (not duplicates): #11531, #2674, #11373.

Expected Behavior

Per the scale-bounds docs,
initial-scale is "the initial target scale a Revision must reach ... before it is marked as Ready."
A new Revision should not become Ready, should not be promoted to
Configuration.status.latestReadyRevisionName, and should not receive route traffic until it reaches
its initial-scale. Traffic should stay on the previous, fully-scaled Revision until then.

Actual Behavior

A new Revision can be marked Ready=True and latched as latestReadyRevisionName while still far below
its initial-scale, so the Route shifts 100% of traffic to an under-provisioned Revision. Because
latestReadyRevisionName is monotonic, the route never reverts — even after the Revision later flips
Ready=False (e.g. ProgressDeadlineExceeded). In production this abandoned a healthy fully-scaled
Revision for a new one running at ~15% of target replicas, which then returned 503/504s.

Root cause (same on release-1.20 and main):

Revision readiness is not gated on initial-scale —
pkg/apis/serving/v1/revision_lifecycle.go:
revisionCondSet = apis.NewLivingConditionSet(ResourcesAvailable, ContainerHealthy) (no
ScaleTargetInitialized/Active term).
ResourcesAvailable comes from the Deployment's Progressing condition, not Available —
pkg/reconciler/revision/reconcile_resources.go PropagateDeploymentStatus →
TransformDeploymentStatus reads {Progressing, ReplicaSetReady}, never appsv1.DeploymentAvailable.
So ResourcesAvailable=True at any replica count, and ContainerHealthy=True once ReadyReplicas > 0.

The only step that holds Ready back below initial-scale (PropagateAutoscalerStatus in reconcilePA,
which pulls ResourcesAvailable back to Unknown while !IsScaleTargetInitialized()) is skipped
when an earlier phase errors. The reconcile is an early-return phase loop
(pkg/reconciler/revision/revision.go):

for _, phase := range []func(context.Context, *v1.Revision) error{
    c.reconcileDeployment,   // sets ResourcesAvailable=True (Progressing) + ContainerHealthy=True
    c.reconcileImageCache,   // if this errors, the loop returns...
    c.reconcilePA,           // ...so reconcilePA (which would reset ResourcesAvailable) is SKIPPED
} {
    if err := phase(ctx, rev); err != nil { return err }
}

A transient reconcileImageCache error (e.g. createImageCache AlreadyExists from image-lister lag,
common under heavy concurrent-write churn) persists the below-initial-scale Ready=True. Then
pkg/reconciler/configuration/configuration.go findAndSetLatestReadyRevision advances
latestReadyRevisionName to it and never reverts (monotonic); the Route follows.
(Note: a reconcileDeployment Update conflict does NOT trigger this — it returns before
PropagateDeploymentStatus, so ResourcesAvailable is never set True that cycle.)

Steps to Reproduce the Problem

In production this fires stochastically under churn. To make it deterministic, the steps below
(a) wedge a new Revision below initial-scale and (b) force reconcileImageCache to error via a
ResourceQuota (a stand-in for the transient AlreadyExists). Requires image caching enabled
(caching.internal.knative.dev Image CRD + controller).

Label 3 schedulable nodes and create a namespace:
kubectl label node repro=true
kubectl create ns repro
Deploy a healthy baseline Revision at scale 1, pinned to the labeled nodes; wait until Ready:
apiVersion: serving.knative.dev/v1
kind: Service
metadata: { name: rt, namespace: repro }
spec:
template:
metadata:
annotations:
autoscaling.knative.dev/initial-scale: "1"
autoscaling.knative.dev/min-scale: "1"
autoscaling.knative.dev/max-scale: "1"
spec:
nodeSelector: { repro: "true" }
containers:
- image: ghcr.io/knative/helloworld-go:latest
env: [{ name: TARGET, value: "v1" }]
Cap the namespace's Image count at its current value so the next Revision's image cache can't be created:
kubectl -n repro create quota capimg
--hard=count/images.caching.internal.knative.dev=$(kubectl -n repro get images.caching.internal.knative.dev --no-headers | wc -l)
Roll a new Revision that wants initial-scale=6 but can only schedule 2 pods (one per labeled node;
one node is held by the baseline) — wedging it at 2/6:

re-apply the same Service with:

spec:
template:
metadata:
annotations:
autoscaling.knative.dev/initial-scale: "6"
autoscaling.knative.dev/min-scale: "6"
autoscaling.knative.dev/max-scale: "6"
spec:
nodeSelector: { repro: "true" }
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- topologyKey: kubernetes.io/hostname
labelSelector:
matchLabels: { serving.knative.dev/service: rt }
containers:
- image: ghcr.io/knative/helloworld-go:latest
env: [{ name: TARGET, value: "v2" }] # forces a new Revision
5. Observe the new Revision become Ready=True and capture the route while below initial-scale:
kubectl -n repro get revision # new rev READY=True
kubectl -n repro get pods # 2 Running, 4 Pending (wedged at 2/6)
kubectl -n repro get images.caching.internal.knative.dev # only the baseline's image exists
kubectl -n repro get configuration rt -o jsonpath='{.status.latestReadyRevisionName}' # -> new rev
kubectl -n repro get route rt -o jsonpath='{.status.traffic}' # -> 100% new rev

The new Revision is Ready=True at 2/6 replicas with no image cache; latestReadyRevisionName and the
Route both point at it, and never revert to the fully-scaled baseline.

Suggested fix

Make readiness reflect replica sufficiency: include the Deployment's Available condition in
TransformDeploymentStatus, or gate Revision Ready / findAndSetLatestReadyRevision advancement on the
PodAutoscaler's ScaleTargetInitialized (enforcing the documented initial-scale semantics). Either
prevents latestReadyRevisionName from advancing to an under-scaled Revision.

Script I used to reproduce the premature route switch.

It doesn't seem to happen all the time, so far I have been able to reproduce it ~4/5 attempts with the script.

#!/usr/bin/env bash
#
# reproduce-early-route-capture.sh  (v8 — mechanism B: below-initial-scale route capture)
#
# Reproduces the exact routing failure behind the 2026-06-16 teeth-segmentation outage: on an
# image bump, the KServe/Knative Route promotes 100% of traffic to the NEW revision while that
# revision is STILL BELOW its initial-scale (prod: ~3 ready of 20). The under-scaled, overwhelmed
# revision then 504s under the model's tight 60s timeout while the healthy prior revision — which
# still had all its replicas — is abandoned. This is the "route points to a revision that is not
# fully scaled up yet" bug, reproduced DETERMINISTICALLY.
#
# ── Mechanism, traced in knative/serving release-1.20 (your cluster's version) ──
#   The Revision reconciler runs a phase loop with early-return-on-error (revision.go):
#       [ reconcileDeployment, reconcileImageCache, reconcilePA ]
#     reconcileDeployment  -> PropagateDeploymentStatus sets ResourcesAvailable=True from the
#                             Deployment's *Progressing* status (NOT Available; reconcile_resources.go),
#                             and ReadyReplicas>0 sets ContainerHealthy=True.
#     reconcileImageCache  -> createImageCache(); if it returns an error the loop RETURNS, so the next
#                             phase (reconcilePA) is SKIPPED.
#     reconcilePA          -> PropagateAutoscalerStatus, which is the ONLY thing that pulls
#                             ResourcesAvailable back to Unknown while below initial-scale
#                             (!ScaleTargetInitialized). When it's skipped, the partial-scale
#                             Ready=True (ResourcesAvailable ∧ ContainerHealthy) PERSISTS.
#   Revision Ready = NewLivingConditionSet(ResourcesAvailable, ContainerHealthy) (revision_lifecycle.go)
#   -- it is NOT gated on initial-scale. So a skipped reconcilePA => Revision Ready=True at, say, 2/6 =>
#   Configuration.latestReadyRevisionName latches it (configuration.go, MONOTONIC, never reverts) =>
#   the Route (traffic latestRevision:true) follows it to the under-scaled revision and never moves back.
#
# ── Why this is a RACE in prod but DETERMINISTIC here ──
#   In prod the reconcileImageCache error was a transient createImageCache "AlreadyExists" (image-lister
#   lag) that only happened because 33 simultaneous rollouts + GPU-stockout scheduling churn hammered
#   the controllers/apiserver with concurrent writes. A quiet cluster never produces that conflict.
#   We induce the SAME createImageCache failure deterministically and gently — a ResourceQuota that caps
#   the Image count in the throwaway namespace, so the new revision's Image simply cannot be created.
#   Same code path, same latch; no load on shared infra. (To instead watch the natural race, you'd need
#   prod-scale concurrent-write pressure — not reproducible from a quiet cluster.)
#
#   Under-scale is made deterministic too: pin both revisions to N_NODES labeled nodes with one-pod-per-
#   node anti-affinity, so the new revision wedges at exactly (N_NODES-1) ready, below initial-scale.
#
# GPU-free. SAFE: throwaway namespace + temporary node labels, both removed on exit.
#
# Exit codes:  0 = PASS   1 = NO-REPRO   2 = INCONCLUSIVE (env)
#
# Usage:  CONTEXT=<ctx> ./reproduce-early-route-capture.sh
#   env: NS IMAGE INITIAL_SCALE N_NODES CPU POLL_TIMEOUT EXPECT KEEP
#        INITIAL_SCALE=6  -> min==max==initial-scale of the rev (mirror prod teeth_seg = 20)
#        N_NODES=3        -> labeled nodes; new rev wedges at (N_NODES-1) ready (must be >=3 -> >=2 ready)
#        EXPECT=captured (default) -> PASS when the route is promoted below initial-scale (pre-fix behavior)
#        EXPECT=held               -> PASS when the route holds on the prior rev (post-fix guard:
#                                     e.g. gate latestReadyRevision on ScaleTargetInitialized, or canaryTrafficPercent)
#        KEEP=oncapture|1          -> keep ns + labels for inspection IF reproduced; KEEP=always -> never tear down
set -uo pipefail   # NOT -e: a transient control-plane blip must not abort a long poll

CONTEXT="${CONTEXT:-$(kubectl config current-context)}"
NS="${NS:-route-capture-repro-$RANDOM}"
IMAGE="${IMAGE:-python:3.11-alpine}"
INITIAL_SCALE="${INITIAL_SCALE:-6}"        # min == max == initial-scale (mirror prod teeth_seg: set to 20)
N_NODES="${N_NODES:-3}"                     # new rev wedges at (N_NODES-1) ready; need >=3 so ready>=2
CPU="${CPU:-0.1}"
POLL_TIMEOUT="${POLL_TIMEOUT:-180}"
EXPECT="${EXPECT:-captured}"
KEEP="${KEEP:-}"
LABEL_KEY="route-capture-repro"
k() { kubectl --context "$CONTEXT" "$@"; }
# resilient read-only query: retries transient control-plane blips, never aborts the script
kq() { local o; for _ in 1 2 3 4; do if o="$(kubectl --context "$CONTEXT" "$@" 2>/dev/null)"; then printf '%s' "$o"; return 0; fi; sleep 2; done; printf ''; return 0; }

NODES=(); _keep=0; [ "$KEEP" = "always" ] && _keep=1
cleanup() {
  if [ "$_keep" = "1" ]; then
    echo "── KEEPING resources (KEEP=$KEEP) ──"
    echo "  inspect : kubectl --context $CONTEXT -n $NS get isvc,route,revision,resourcequota,pods -o wide"
    echo "  teardown: kubectl --context $CONTEXT delete ns $NS"
    [ -n "${NODES[*]:-}" ] && echo "            for n in ${NODES[*]}; do kubectl --context $CONTEXT label node \$n ${LABEL_KEY}-; done"
    return
  fi
  echo "── cleanup ──"
  k delete ns "$NS" --wait=false >/dev/null 2>&1 || true
  for n in "${NODES[@]:-}"; do k label node "$n" "${LABEL_KEY}-" >/dev/null 2>&1 || true; done
}
trap cleanup EXIT

echo "context: $CONTEXT   expect: $EXPECT   initial-scale: $INITIAL_SCALE   nodes: $N_NODES (-> new rev wedged at $((N_NODES-1))/$INITIAL_SCALE)"
[ "$N_NODES" -ge 3 ] || { echo "ERROR: N_NODES must be >=3 (need >=2 ready on the new rev)"; exit 2; }
[ "$INITIAL_SCALE" -gt "$((N_NODES-1))" ] || { echo "ERROR: INITIAL_SCALE must exceed N_NODES-1 (else not under-scaled)"; exit 2; }
k get ns knative-serving >/dev/null 2>&1 || { echo "ERROR: knative-serving not found"; exit 2; }
k get ns kserve >/dev/null 2>&1 || k get ns kserve-system >/dev/null 2>&1 || { echo "ERROR: kserve not found"; exit 2; }
k get crd images.caching.internal.knative.dev >/dev/null 2>&1 || { echo "ERROR: image caching CRD absent; the deterministic trigger needs it"; exit 2; }

echo "── selecting $N_NODES non-GPU schedulable nodes ──"
cat > /tmp/.route-repro-pick.py <<PY
import json,subprocess,sys
ctx="$CONTEXT"; want=int("$N_NODES")
nodes=json.loads(subprocess.check_output(["kubectl","--context",ctx,"get","nodes","-o","json"]))["items"]
out=[]
for n in nodes:
    a=n["status"].get("allocatable",{})
    if a.get("nvidia.com/gpu") not in (None,"0",0): continue
    if any(t.get("effect")=="NoSchedule" for t in n["spec"].get("taints",[])): continue
    if n["spec"].get("unschedulable"): continue
    out.append(n["metadata"]["name"])
for nm in out[:want]: print(nm)
PY
mapfile -t NODES < <(python3 /tmp/.route-repro-pick.py)
[ "${#NODES[@]}" -ge "$N_NODES" ] || { echo "ERROR: found ${#NODES[@]} schedulable non-GPU nodes, need $N_NODES."; exit 2; }
for n in "${NODES[@]}"; do echo "  pin $n"; k label node "$n" "${LABEL_KEY}=true" --overwrite >/dev/null; done

k create ns "$NS" >/dev/null 2>&1 || true
k label ns "$NS" istio-injection=enabled --overwrite >/dev/null 2>&1 || true

# trivial fast server: /ready passes ~2s after start; / returns 200 identifying the revision.
read -r -d '' SERVER_PY <<'PY' || true
import time, http.server, socketserver, os
START = time.time(); TARGET = os.environ.get("TARGET", "?")
class H(http.server.BaseHTTPRequestHandler):
    def log_message(self, *a): pass
    def do_GET(self):
        if self.path.startswith("/ready"):
            ok = (time.time() - START) >= 2
            self.send_response(200 if ok else 503); self.end_headers(); self.wfile.write(b"ok" if ok else b"x"); return
        self.send_response(200); self.end_headers(); self.wfile.write(("served-by=" + TARGET + "\n").encode())
class S(socketserver.ThreadingMixIn, http.server.HTTPServer): daemon_threads = True
S(("0.0.0.0", 8080), H).serve_forever()
PY
SERVER_B64="$(printf '%s' "$SERVER_PY" | base64 | tr -d '\n')"

# $1=TARGET  $2=scale(min==max==initial-scale)  $3=antiAffinity(0|1)
isvc_yaml() {
  local aff="" ann=""
  [ "$3" = 1 ] && aff="
    affinity: {podAntiAffinity: {requiredDuringSchedulingIgnoredDuringExecution: [{topologyKey: kubernetes.io/hostname, labelSelector: {matchLabels: {serving.knative.dev/configuration: rt-predictor}}}]}}"
  [ "$2" -gt 1 ] && ann="
    autoscaling.knative.dev/initial-scale: \"$2\""   # chart sets initial-scale == minReplicas when min > 1
  cat <<EOF
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: rt
  namespace: ${NS}
  labels: {sidecar.istio.io/inject: "true"}
  annotations: {${ann}}
spec:
  predictor:
    minReplicas: $2
    maxReplicas: $2
    containerConcurrency: 1
    nodeSelector: {${LABEL_KEY}: "true"}
    containers:
      - name: kserve-container
        image: ${IMAGE}
        ports: [{containerPort: 8080, protocol: TCP}]
        command: ["/bin/sh","-c"]
        args: ["echo ${SERVER_B64} | base64 -d > /tmp/s.py && exec python3 /tmp/s.py"]
        env: [{name: TARGET, value: "$1"}]
        readinessProbe: {httpGet: {path: /ready, port: 8080}, initialDelaySeconds: 1, periodSeconds: 2, failureThreshold: 90}
        resources: {requests: {cpu: "${CPU}"}, limits: {cpu: "${CPU}"}}$aff
EOF
}

curl_route() {
  k -n "$NS" run "c$RANDOM" --image=curlimages/curl --restart=Never --attach --rm -q --command -- \
    sh -c "curl -s -m 8 -w ' %{http_code}' http://rt.${NS}.svc.cluster.local/ 2>/dev/null | tr -d '\n' || echo ' 000'" 2>/dev/null
}

# rev1: healthy baseline (scale 1) — takes one node, creates its Image, serves 200.
echo "── rev1 baseline ──"
isvc_yaml rev1 1 0 | k apply -f - >/dev/null
if ! k -n "$NS" wait --for=condition=Ready isvc/rt --timeout=240s >/dev/null 2>&1; then
  echo "INCONCLUSIVE: rev1 baseline didn't become Ready"; exit 2
fi
echo "  rev1 ready; route serves [$(curl_route) ]"

# Cap the Image count at its current value so the NEW revision's createImageCache fails deterministically
# -> reconcileImageCache errors every reconcile -> reconcilePA is skipped -> the below-scale Ready latches.
imgcount="$(kq -n "$NS" get images.caching.internal.knative.dev --no-headers | grep -c .)"; imgcount="${imgcount:-1}"
echo "── capping count/images.caching.internal.knative.dev at $imgcount (forces new-rev image-cache failure) ──"
cat <<EOF | k apply -f - >/dev/null
apiVersion: v1
kind: ResourceQuota
metadata: {name: capimg, namespace: ${NS}}
spec:
  hard: {count/images.caching.internal.knative.dev: "${imgcount}"}
EOF

# rev2: the rollout — wants INITIAL_SCALE, anti-affinity wedges it at (N_NODES-1) ready, and its Image
# can't be created (quota) so reconcileImageCache keeps erroring -> reconcilePA skipped -> latch.
echo "── rev2 rollout (wedged below initial-scale; image-cache create blocked) ──"
isvc_yaml rev2 "$INITIAL_SCALE" 1 | k apply -f - >/dev/null
NEWREV=""; d=$(( $(date +%s) + 60 ))
while [ "$(date +%s)" -lt "$d" ]; do
  NEWREV="$(kq -n "$NS" get isvc rt -o jsonpath='{.status.components.predictor.latestCreatedRevision}')"
  [ -n "$NEWREV" ] && [ "$NEWREV" != "rt-predictor-00001" ] && break; sleep 2
done
echo "  baseline=rt-predictor-00001  new=$NEWREV"

# count the new rev's actually-Ready pods directly (revision .status.actualReplicas is populated by
# PropagateAutoscalerStatus — the very step being skipped — so it stays empty; the pods are the truth).
ready_pods() { kq -n "$NS" get pods -l serving.knative.dev/revision="$NEWREV" \
  -o jsonpath='{range .items[*]}{.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}' | grep -c True; }

d=$(( $(date +%s) + POLL_TIMEOUT )); lr=""; rp=0; maxrp=0
while [ "$(date +%s)" -lt "$d" ]; do
  lr="$(kq -n "$NS" get isvc rt -o jsonpath='{.status.components.predictor.latestReadyRevision}')"
  rp="$(ready_pods)"; rp="${rp:-0}"
  [ "$rp" -gt "$maxrp" ] 2>/dev/null && maxrp="$rp"
  printf '\r  latestReady=%s  new ready-pods=%s/%s (peak %s)      ' "$lr" "$rp" "$INITIAL_SCALE" "$maxrp"
  [ "$lr" = "$NEWREV" ] && break
  [ "$rp" -ge "$INITIAL_SCALE" ] 2>/dev/null && break
  sleep 2
done
echo

newready="$(kq -n "$NS" get revision "$NEWREV" -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}')"
route="$(kq -n "$NS" get route rt-predictor -o jsonpath='{range .status.traffic[*]}{.revisionName}={.percent}% {end}')"
imgs="$(kq -n "$NS" get images.caching.internal.knative.dev -l serving.knative.dev/revision="$NEWREV" --no-headers | grep -c .)"
served="$(curl_route)"
echo "  new ready-pods=$maxrp/$INITIAL_SCALE  newRevReady=$newready  latestReady=$lr  new-rev images=$imgs (quota-blocked)"
echo "  route=[$route]  served-through-route=[$served ]"

echo "════════════════════════════════════════"
# CAPTURE (the bug): route latched to the new rev while it has FEWER than initial-scale ready pods.
if [ "$lr" = "$NEWREV" ] && [ "$maxrp" -lt "$INITIAL_SCALE" ]; then
  case "$KEEP" in oncapture|1|always) _keep=1;; esac
  echo "OUTCOME: route promoted to $NEWREV with only ${maxrp}/${INITIAL_SCALE} ready pods (Revision Ready=$newready, no image cache)."
  echo "         latestReadyRevision advanced below initial-scale and the Route follows it — the under-scaled-capture bug."
  echo "         (served-through-route=$served: a request to the under-scaled rev — 503/504/non-200 = the abandoned-capacity failure.)"
  [ "$EXPECT" = "captured" ] && { echo "PASS (expected captured)"; exit 0; } || { echo "FAIL: expected 'held' (fix regressed?)"; exit 1; }
fi

if [ "$maxrp" -ge "$INITIAL_SCALE" ]; then
  echo "OUTCOME: new rev reached full initial-scale (anti-affinity cap failed? NEWREV=$NEWREV) — under-scale not demonstrated."
else
  echo "OUTCOME: route did NOT promote to $NEWREV within ${POLL_TIMEOUT}s (latestReady=$lr); it kept serving the prior revision."
fi
[ "$EXPECT" = "held" ] && { echo "PASS (expected held — route never moved below initial-scale)"; exit 0; } || { echo "NO-REPRO"; exit 1; }

kserve/kserve#5683

Do you have a repro without using KServe and just vanilla Knative?

No... not at the moment

@dprotaso

@dprotaso and @juanpark-dandy I'm working on reproducing this locally before proposing a fix. I hope this is welcome!

@juanpark-dandy

UPDATE: I reproduced this with vanilla knative/serving v1.22.1 on a disposable three-node Kind cluster. The commands below create the environment from public release manifests.

While I believe the bug can be solved with the suggested fix proposed by @juanpark-dandy, I wonder if this issue is linked to issue #5076 and is thus a part of a bigger coordination issue that deserves a more holistic solution. I won't be opposed to any patches in the meantime.

Setup

Creating the 3 node cluster and installing Knative/serving + Kourier:

set -euo pipefail

REPRO_CLUSTER=issue-16649
REPRO_DIR=/tmp/issue-16649
REPRO_KUBECONFIG="${REPRO_DIR}/kubeconfig"

docker info >/dev/null
kind get clusters # must not contain "issue-16649"

mkdir -p "${REPRO_DIR}"
cat >"${REPRO_DIR}/kind.yaml" <<'EOF'
apiVersion: kind.x-k8s.io/v1alpha4
kind: Cluster
nodes:
  - role: control-plane
  - role: worker
  - role: worker
EOF

kind create cluster \
  --name "${REPRO_CLUSTER}" \
  --image kindest/node:v1.34.8@sha256:02722c2dedddcfc00febf5d27fbeb9b7b2c14294c82109ff4a85d89ac9ba3256 \
  --config "${REPRO_DIR}/kind.yaml" \
  --kubeconfig "${REPRO_KUBECONFIG}"

export KUBECONFIG="${REPRO_KUBECONFIG}"
test "$(kubectl config current-context)" = "kind-${REPRO_CLUSTER}"

kubectl wait --for=condition=Ready nodes --all --timeout=180s
kubectl taint nodes \
  -l node-role.kubernetes.io/control-plane \
  node-role.kubernetes.io/control-plane-

kubectl apply -f https://github.com/knative/serving/releases/download/knative-v1.22.1/serving-crds.yaml
kubectl apply -f https://github.com/knative/serving/releases/download/knative-v1.22.1/serving-core.yaml
kubectl apply -f https://github.com/knative-extensions/net-kourier/releases/download/knative-v1.22.1/kourier.yaml

kubectl -n knative-serving patch configmap config-network --type=merge \
  -p '{"data":{"ingress-class":"kourier.ingress.networking.knative.dev"}}'
kubectl -n knative-serving patch configmap config-features --type=merge \
  -p '{"data":{"kubernetes.podspec-affinity":"enabled"}}'

kubectl -n knative-serving wait \
  --for=condition=Available deployment --all --timeout=300s
kubectl -n kourier-system wait \
  --for=condition=Available deployment --all --timeout=300s

The node table must show three Ready=True nodes with TAINTS=<none>:

kubectl get nodes -o custom-columns='NAME:.metadata.name,READY:.status.conditions[?(@.type=="Ready")].status,TAINTS:.spec.taints'
kubectl get crd images.caching.internal.knative.dev
kubectl -n knative-serving get deployment/controller -o jsonpath='servingVersion={.metadata.labels.app\.kubernetes\.io/version}{"\n"}'

In a second Zsh terminal:

export KUBECONFIG=/tmp/issue-16649/kubeconfig
test "$(kubectl config current-context)" = kind-issue-16649
kubectl -n kourier-system port-forward service/kourier-internal 18080:80

Creating the baseline Service with scale as 1:

IMAGE=ghcr.io/knative/helloworld-go@sha256:c2b7412fbea6f1ef24a0cac60698e88df7ae3c4278e42d0cb34fe7d4b2641bba
kubectl create namespace repro

kubectl apply -f - <<EOF
apiVersion: serving.knative.dev/v1
kind: Service
metadata: {name: rt, namespace: repro}
spec:
  template:
    metadata:
      annotations: {autoscaling.knative.dev/initial-scale: "1", autoscaling.knative.dev/min-scale: "1", autoscaling.knative.dev/max-scale: "1"}
    spec:
      containers:
        - image: "${IMAGE}"
          env: [{name: TARGET, value: v1}]
EOF

kubectl -n repro wait --for=condition=Ready kservice/rt --timeout=300s

curl --connect-timeout 2 --max-time 10 -sS \
  -w '\nHTTP %{http_code}\n' \
  -H 'Host: rt.repro.svc.cluster.local' \
  http://127.0.0.1:18080

Curl must return Hello v1! and HTTP 200.

Issue Repro

Applying the hardcoded image cap and rolling out a new Service re-creates the issue.

kubectl -n repro create quota cap-image \
  --hard=count/images.caching.internal.knative.dev=1

USED=
for _ in {1..60}; do
  USED="$(kubectl -n repro get quota cap-image \
    -o go-template='{{index .status.used "count/images.caching.internal.knative.dev"}}')"
  [ "${USED}" = 1 ] && break
  sleep 1
done
test "${USED}" = 1


kubectl apply -f - <<EOF
apiVersion: serving.knative.dev/v1
kind: Service
metadata: {name: rt, namespace: repro}
spec:
  template:
    metadata:
      annotations: {autoscaling.knative.dev/initial-scale: "6", autoscaling.knative.dev/min-scale: "6", autoscaling.knative.dev/max-scale: "6"}
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - topologyKey: kubernetes.io/hostname
              labelSelector:
                matchLabels: {serving.knative.dev/service: rt}
      containers:
        - image: "${IMAGE}"
          env: [{name: TARGET, value: v2}]
EOF

kubectl -n repro wait \
  --for=create revision/rt-00002 --timeout=60s
kubectl -n repro wait \
  --for='jsonpath={.status.conditions[?(@.type=="Ready")].status}=True' \
  revision/rt-00002 --timeout=90s
kubectl -n repro wait \
  --for='jsonpath={.status.latestReadyRevisionName}=rt-00002' \
  configuration/rt --timeout=90s
kubectl -n repro wait \
  --for='jsonpath={.status.traffic[0].revisionName}=rt-00002' \
  route/rt --timeout=90s

These commands show the partial reconciliation with Kourier returning Hello v1! after the failed v2 rollout:

kubectl -n repro get revision,configuration,route,deployment,podautoscaler,image,pod -o wide
kubectl -n repro get ingress.networking.internal.knative.dev/rt \
  -o jsonpath='ingressBackend={.spec.rules[0].http.paths[0].splits[0].serviceName}{"\n"}'

curl --connect-timeout 2 --max-time 10 -sS \
  -w '\nHTTP %{http_code}\n' \
  -H 'Host: rt.repro.svc.cluster.local' \
  http://127.0.0.1:18080

rt-00002 has two ready pods out of six while its Revision reported Ready=True. Configuration and Route status named rt-00002, with Ingress on rt-00001 because the injected Image error was still active. The v2 Image and PodAutoscaler are absent. This independently reproduces the partial-reconciliation/latest-ready latch with vanilla Knative

Thanks for the thorough root-cause analysis and the vanilla repro — that's a solid diagnosis.

I'd like to take a crack at this. I agree with the mechanism: Revision Ready is just ResourcesAvailable && ContainerHealthy, ResourcesAvailable is derived from the Deployment's Progressing condition rather than Available, and the phase loop's early-return on a reconcileImageCache error skips reconcilePA — the only step that pulls ResourcesAvailable back to Unknown while !IsScaleTargetInitialized(). So a below-initial-scale Revision can latch Ready=True and findAndSetLatestReadyRevision advances monotonically.

Of the two directions suggested, I'd lean toward the one that enforces the documented contract at the point of latching: gate findAndSetLatestReadyRevision on the Revision's PodAutoscaler having reached ScaleTargetInitialized (i.e. don't advance latestReadyRevisionName to a Revision that hasn't reached its initial-scale). That covers the image-cache-error path too, since it's the PA status that never got propagated there, rather than only fixing the deployment-condition path.

Rough shape:

In pkg/reconciler/configuration/configuration.go, require the candidate Revision to report ScaleTargetInitialized=True (from the PodAutoscaler propagated into Revision.status) before advancing latestReadyRevisionName.
Unit test: a Revision with Ready=True but ScaleTargetInitialized unset/Unknown must not be latched.
Envtest case mirroring the quota-triggered repro in this thread, asserting latestReadyRevisionName (and therefore the Route) stays on the baseline Revision.

I'm not wedded to that over making TransformDeploymentStatus consume the Deployment's Available condition — the gating approach just seems smaller and closer to the "initial-scale before Ready" semantics. If this direction looks right I'll open a PR; happy to leave it if you'd rather handle it or prefer the other approach.

Navigation Menu

Repository navigation

Route promotes a Revision to 100% traffic while it is still below initial-scale (latestReadyRevision latches an under-scaled Revision) #16649

What version of Knative?

Expected Behavior

Actual Behavior

re-apply the same Service with:

Setup

Issue Repro

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Route promotes a Revision to 100% traffic while it is still below initial-scale (latestReadyRevision latches an under-scaled Revision) #16649

Description

What version of Knative?

Expected Behavior

Actual Behavior

re-apply the same Service with:

Activity

juanpark-dandy commented on Jun 17, 2026

juanpark-dandy commented on Jun 17, 2026

juanpark-dandy commented on Jun 17, 2026

dprotaso commented on Jun 17, 2026

juanpark-dandy commented on Jun 19, 2026

adityasrini commented on Jul 22, 2026

adityasrini commented on Jul 30, 2026

Setup

Issue Repro

shashankvarma499 commented on Sep 13, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions