feat: WorkerPool autoscaling — demand-reactive capacity for warm worker pools

Summary

WorkerPool replicas are static today: a human sets spec.replicas and the controller applies it verbatim. There is no mechanism that grows or shrinks a pool in response to actor demand. This issue opens a discussion on an autoscaling story for WorkerPools — what it should optimize for, why "just attach an HPA" is not sufficient for substrate's workload shape, and a concrete approach we've been exploring (a warm-buffer model with a fast scale-up trigger and a slow scale-down). The goal here is to align on direction and on the in-tree vs. out-of-tree boundary.

Autoscaling is already named as future work in docs/roadmap.md ("Worker horizontal autoscaling: Ability to rapidly scale up nodes and warm Pods to meet actor demand") and docs/architecture.md ("Autoscaling: We will need to be able to automatically scale the number of workers up and down based on demand…"), but there's no design or tracking issue for it yet.

Current state

WorkerPoolSpec carries only replicas + ateomImage. No min/max bounds, no target utilization, no metric, no demand input.
The scale subresource is exposed: the CRD declares
+kubebuilder:subresource:scale:specpath=.spec.replicas,statuspath=.status.replicas,
but HPA integration is incomplete; no pool-occupancy/capacity metrics exist
A full pool fails fast: when no idle worker exists, AssignWorkerStep returns FailedPrecondition "no free workers available", which the router maps to HTTP 503. There is no queue/wait (tracked separately in Router needs to park requests and wait for capacity #27) and no capacity creation in response.

Net: the substrate partially provides the lever (the scale subresource) but no control loop and partial signals to drive one.

Why autoscaling matters

The density premise requires elasticity. Substrate's whole value is multiplexing many actors onto few warm workers. Without autoscaling, operators must statically over-provision for peak (wasting money on idle workers) or under-provision (and serve 504 timeouts under load). Elastic pools are what let density translate into actual cost savings.
Cost — scale-to-zero / scale-to-floor for idle pools. Idle actors are just snapshots in object storage; a pool serving no active actors is pure waste. Autoscaling (down to a small floor, or to zero for cold-start-tolerant pools) directly reduces spend.
Activation-latency SLO under bursts. Agent workloads tend to wake in correlated storms (sub-agent tasks/CI fan-out). Maintaining warm headroom and refilling it reactively is what helps the storm pass.
Eliminating avoidable 503s. Today an empty pool returns 503 with no recourse. Pairing capacity creation with the request-parking work in Router needs to park requests and wait for capacity #27 turns "no capacity → error" into "no capacity → brief wait → served."
Operability. Operators shouldn't hand-tune replicas per pool per time-of-day. A control loop (with sane bounds) mandatory for running this in production.

The core constraint - why a naive metric→HPA isn't enough

The binding latency is pod start (schedule + image pull + ateom boot + readiness — seconds to minutes; plus node provisioning if the cluster is full).

Pure metric-loop reactivity is bounded by detection lag. Prometheus scrape (~15–30s) + KEDA/HPA poll & stabilization (tens of seconds) is perfectly fine for smooth/diurnal demand, but too slow as the sole up-path for spiky wake-storms with a lean buffer — the buffer empties and stays empty through the lag window.

So "reactive enough" should be read as "fast enough to refill the buffer before the next comparable burst," not "fast enough to serve this burst."

Proposed approach (for discussion)

A warm-buffer invariant, replenished on consumption

Each pool targets a small number of idle (warm) workers. When a resume consumes a slot and the buffer dips below target, emit a refill — sized net of in-flight provisioning (anti-windup), so the loop doesn't pile on scale-ups while pods are still booting:

deficit = max(0, target - (free + in_flight)) // buffer refill
scale_to = clamp(current + deficit, reservation_floor, max_replicas).  // rate-limited per step

When preemption becomes available, preemptible actors will be deducted from the deficit.

Where should this live?

Substrate is intentionally low-opinion, so the policy could reasonably live outside the core. Much of the above is buildable today as an external controller — it can drive the existing scale subresource and read pool occupancy via a new WatchWorkers API.

Scale-to-zero and node headroom

Scale-to-zero is attractive for idle/cold-start-tolerant pools, but the first post-zero request pays full cold start; keep a per-pool floor for latency-sensitive pools - Will handle in a follow-up issue
Pod scale-up only helps if a node has room. A pod request that lands Pending behind cluster-autoscaler/Karpenter is not fast. Warm node headroom is a complementary lever worth calling out.

Update 2: The below approach has been dropped, the main issue was updated with the new approach
Update 1: dropping the stock-HPA down-path (re: open question 1)

When I opened this I assumed scale-down would be a stock metric loop (HPA/KEDA on the scale subresource), with a separate fast trigger only for scale-up. Building it out, I dropped HPA and folded both directions into a single in-tree autoscaler. Why:

Single-writer integrity — the decisive one. The original issue specifies spec.replicas has exactly one writer, for the fast up-path and the slow down-path. Creating a component that synchronizes pressure events from ateapi, and scale down events from an extern autoscaler is awkward (e.g. the trigger bumps the HPA's minReplicas instead of /scale) and still splits one control law across two systems. One component owning both directions makes single-writer structural instead of a coordination problem.
Fewer dependencies, no detection lag. No HPA/KEDA/Prometheus to install or operate. The loop reads occupancy straight from ateapi (ListWorkers) — no scrape lag.
What it is now: a single WorkerPoolAutoscaler in ate-controller, the sole writer of spec.replicas. Up-path = event-driven capacity-pressure from ateapi (new WatchCapacityPressure stream, emitted where the resume misses). Down-path = the same loop's hysteretic poll on occupancy (which doubles as the safety net). minReady / targetBuffer / maxReplicas are first-class WorkerPoolSpec fields (re: open question 2).

flowchart TD
  subgraph dp["Data plane · ateapi"]
    MISS["Resume finds no free worker<br/>(AssignWorkerStep — t=0)"]
    OCC["Per-pool occupancy<br/>free / in-flight (ListWorkers)"]
  end
  MISS -->|"capacity-pressure event<br/>WatchCapacityPressure · fast"| AS
  OCC -->|"reconcile poll ~10s<br/>hysteretic · safety net"| AS
  subgraph ctl["ate-controller · single writer of spec.replicas"]
    AS["WorkerPoolAutoscaler 
UP immediate
 DOWN after stabilization"]
  end
  AS -->|"patch /scale"| WP["WorkerPool.spec.replicas"]
  WP --> WPR["WorkerPoolReconciler<br/>materializes the Deployment"]
  WPR --> PODS["Warm worker pods<br/>join / drain the buffer"]
  PODS -.->|"register / release"| OCC

feat(api): add WorkerPool autoscaling bounds to WorkerPoolSpec

feat(autoscaler): add pure WorkerPool autoscaling decision logic

feat(autoscaler): add WorkerPool autoscaler reconciler

feat(ateapi): emit capacity-pressure signal on resume miss

feat(autoscaler): trigger reactive scale-up on capacity pressure

docs(demo): add WorkerPool autoscaling demo

feat(api): add WorkerPool autoscaling bounds to WorkerPoolSpec

feat(autoscaler): add pure WorkerPool autoscaling decision logic

feat(autoscaler): add WorkerPool autoscaler reconciler

feat(ateapi): emit capacity-pressure signal on resume miss

feat(autoscaler): trigger reactive scale-up on capacity pressure

Thanks Omer! As a first milestone, I'm adding a demo on how to autoscale workerpools with a basic HPA, so we can experiment with autoscaling, even if too slow for now: #576

We've found some problems with autoscaling that still need to be addressed for a good experience, for example:

Scale-down can strand paused actors. When a worker pod is removed, a local-snapshot paused actor keeps its node pin pointing at a node that may no longer have (free) worker pods.
- If scale-down removes the last worker on that node, the actor is stuck at no free workers available until a new worker is created on that node.
Scale-up gives no locality guarantee. A new worker pod may land on a "useless" node.
- If there are multiple local-snapshot paused actors that can't resume because no worker pods are available on their node, upscaling will not unblock them: the added capacity can land anywhere, so the pool can grow without ever placing a worker where a stranded actor needs it.
No guardrails on worker deletion. There is no pod finalizer holding a pod in Terminating until its actors are safely snapshotted/paused, so deleting a worker abruptly interrupts whatever it was running.
- Tracked in issue #23.
Scale-down picks victims blind to actor assignment (can kill busy pods). A WorkerPool is backed by a Deployment, so shrinking replicas hands victim selection to the built-in Deployment → ReplicaSet controller.
- That controller has no knowledge of which pods have actors assigned → it chooses which pods to delete by generic Kubernetes heuristics.
- So on scale down it can delete a busy pod (one running an actor) while keeping an idle one, purely by chance. Nothing biases deletion toward free workers, because worker assignment lives in the ateapi store, not on the pod object the ReplicaSet controller reasons about.

Luiz Oliveira (@laoj2)

Thanks Luiz Oliveira (@laoj2) !

Thanks Omer! As a first milestone, I'm adding a demo on how to autoscale workerpools with a basic HPA, so we can experiment with autoscaling, even if too slow for now: #576

We've found some problems with autoscaling that still need to be addressed for a good experience, for example:

Scale-down can strand paused actors. When a worker pod is removed, a local-snapshot paused actor keeps its node pin pointing at a node that may no longer have (free) worker pods.

If scale-down removes the last worker on that node, the actor is stuck at no free workers available until a new worker is created on that node.

Scale-up gives no locality guarantee. A new worker pod may land on a "useless" node.

If there are multiple local-snapshot paused actors that can't resume because no worker pods are available on their node, upscaling will not unblock them: the added capacity can land anywhere, so the pool can grow without ever placing a worker where a stranded actor needs it.

Those are interesting issues, I'd love to have a chat with you and Julian Gutierrez Oschmann (@juli4n) to discuss them

No guardrails on worker deletion. There is no pod finalizer holding a pod in Terminating until its actors are safely snapshotted/paused, so deleting a worker abruptly interrupts whatever it was running.

Tracked in issue #23.

Scale-down picks victims blind to actor assignment (can kill busy pods). A WorkerPool is backed by a Deployment, so shrinking replicas hands victim selection to the built-in Deployment → ReplicaSet controller.

That controller has no knowledge of which pods have actors assigned → it chooses which pods to delete by generic Kubernetes heuristics.

So on scale down it can delete a busy pod (one running an actor) while keeping an idle one, purely by chance. Nothing biases deletion toward free workers, because worker assignment lives in the ateapi store, not on the pod object the ReplicaSet controller reasons about.

WRT the last two issues, I am working on my implementation ATM, and I have planned to integrate Pod Deletion Cost to try to prevent deletion of workers with running actors. It gives no guarantees, but it could complementary to all the solutions of the above issues

I'll try to have my implementation PR updated sometime tomorrow and get your opinions on it, and will also take a look at your PR

Add autoscaled-workerpool demo using HPA and prometheus-adapter on Ki…

Luiz Oliveira (@laoj2)

A few notes from recent conversations:

Luiz Oliveira (@laoj2) and I thought that we might have to write a custom worker-pool controller, but Ron Lev (@ronlv10) had an interesting idea: combining worker drain/taint + Pod Deletion Cost will give us a deterministic way of choosing exactly which worker will be deleted in a scale down operation.
We might not want to introduce new fields to the WorkerPool API, so we should discuss if we want to implement the "faser" autoscaler (described in this issue) as an out-of-tree component with its own API, or adopt existing APIs (HPA, ScaledObject etc) and implement them specifically for substrate in said autoscaler.

The design I am working on now has changed since the inception of this issue:

I want to introduce a new WatchWorkers API to ateapi for the autoscaler to consume
The autoscaler will react to worker assignment/release events and will scale the pool accordingly (instead of listening for pressure events from atenet)

Julian Gutierrez Oschmann (@juli4n) minhngo248

Omer Yahud (@omeryahud)

Omer Yahud (@omeryahud) According to your potential design, I have some ideas

Why don't we merge WatchWorkers API into WorkerPoolAutoscaler API? At the moment, I haven't yet seen the need of external WatchWorkers API
Custom worker pool controller sounds like a great idea. We can also introduce WorkerPoolAutoscaler custom resource.
Here is my suggestion for WorkerPoolAutoscaler API:

apiVersion: minhnn.fr/v1alpha1 # Just an example API version
kind: WorkerPoolAutoscaler
metadata:
  name: wpa
spec:
  workerPoolRef:
    name: some-workerpool
  minReplicas: 1
  maxReplicas: 10
  scaleUpThreshold: 0
  scaleDownThreshold: 2
  stepSize: 1
  pollInterval: 10s

Julian Gutierrez Oschmann (@juli4n)

Thanks for the ideas!

Luiz Oliveira (@laoj2) and I thought that we might have to write a custom worker-pool controller, but Ron Lev (@ronlv10) had an interesting idea: combining worker drain/taint + Pod Deletion Cost will give us a deterministic way of choosing exactly which worker will be deleted in a scale down operation.

That's an interesting approach that could work fine for a custom autoscaler controller. Unfortunately, I think this wouldn't work for HPA, though. Given that we'd need to drain the worker + annotate the pod before the HPA controller patches the scale subresource, which would be racey (see the problems section in kubernetes/kubernetes#123541 too).

I think we'd still need a mechanism to drain/taint workers, though. This would be needed for both HPA+Custom WorkerPool controller and Custom Autoscaler approaches

The design I am working on now has changed since the inception of this issue:

I want to introduce a new WatchWorkers API to ateapi for the autoscaler to consume

The autoscaler will react to worker assignment/release events and will scale the pool accordingly (instead of listening for pressure events from atenet)

Luiz Oliveira (@laoj2)

Thanks for the ideas!

Luiz Oliveira (@laoj2) and I thought that we might have to write a custom worker-pool controller, but Ron Lev (@ronlv10) had an interesting idea: combining worker drain/taint + Pod Deletion Cost will give us a deterministic way of choosing exactly which worker will be deleted in a scale down operation.

That's an interesting approach that could work fine for a custom autoscaler controller. Unfortunately, I think this wouldn't work for HPA, though. Given that we'd need to drain the worker + annotate the pod before the HPA controller patches the scale subresource, which would be racey (see the problems section in kubernetes/kubernetes#123541 too).

I think we'd still need a mechanism to drain/taint workers, though. This would be needed for both HPA+Custom WorkerPool controller and Custom Autoscaler approaches

I actually think this would work for HPA too. I'm not sure if it was clear, but the WorkerPool controller is the component proposed to handle draining and adding the deletion cost.

The flow would look like this: the HPA updates the worker pool scale, triggering a reconcile in the WorkerPool controller, The controller then chooses which workers to remove, drains them, adds the deletion cost, and updates the RS. Luiz Oliveira (@laoj2) WDYT ?

Ron Lev (@ronlv10)

I actually think this would work for HPA too. I'm not sure if it was clear, but the WorkerPool controller is the component proposed to handle draining and adding the deletion cost.

The flow would look like this: the HPA updates the worker pool scale, triggering a reconcile in the WorkerPool controller, The controller then chooses which workers to remove, drains them, adds the deletion cost, and updates the RS. Luiz Oliveira (@laoj2) WDYT ?

Thanks for clarifying Ron Lev (@ronlv10), I had misunderstood the proposal. Yes, I also agree that this should work with a stock HPA as well and have more control on what workers to remove

Julian Gutierrez Oschmann (@juli4n)

Julian Gutierrez Oschmann (@juli4n) Luiz Oliveira (@laoj2) Would appreciate your feedback on the above diagram

Navigation Menu

Repository navigation

feat: WorkerPool autoscaling — demand-reactive capacity for warm worker pools #198

Summary

Current state

Why autoscaling matters

The core constraint - why a naive metric→HPA isn't enough

Proposed approach (for discussion)

A warm-buffer invariant, replenished on consumption

Where should this live?

Scale-to-zero and node headroom

13 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

feat: WorkerPool autoscaling — demand-reactive capacity for warm worker pools #198

Description

Summary

Current state

Why autoscaling matters

The core constraint - why a naive metric→HPA isn't enough

Proposed approach (for discussion)

A warm-buffer invariant, replenished on consumption

Where should this live?

Scale-to-zero and node headroom

Activity

omeryahud commented on Jun 10, 2026

13 remaining items

laoj2 commented on Jul 28, 2026

omeryahud commented on Jul 29, 2026

omeryahud commented on Jul 30, 2026

minhngo248 commented on Jul 30, 2026

laoj2 commented on Jul 31, 2026

omeryahud commented on Jul 31, 2026

ronlv10 commented on Jul 31, 2026

laoj2 commented on Jul 31, 2026

omeryahud commented on Aug 3, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions