Identify what metrics we should use for HPA-based workerpool autoscaling

Today, the HPA demo is configured to autoscale a workerpool based on ate.workerpool.workers metric:

substrate/demos/autoscaled-workerpool/prometheus-adapter.yaml

Lines 47 to 57 in e70e1f2

    
             config.yaml: | 
        
               externalRules: 
        
               - seriesQuery: '{__name__="ate_workerpool_workers"}' 
        
                 resources: 
        
                   namespaced: false 
        
                 name: 
        
                   matches: "^(.*)$" 
        
                   as: "$1" 
        
                 # max() across ate-api-server replicas since each replica reports the 
        
                 # global count. 
        
                 metricsQuery: 'max(<<.Series>>{<<.LabelMatchers>>}) by (ate_workerpool_namespace, ate_workerpool_name, ate_worker_state)'

ate.workerpool.workers counts workers by state, so it's clamped by capacity: eg, when a pool of 5 is fully assigned it reports 5, whether one actor is waiting or a thousand are.

Scaling on this metric alone may be fine for the steady state, but we'll be blind to incoming demand that greatly exceeds the pool capacity.

atenet.router.parking.active is a demand sign, but it doesn't know what specific workerpool the demand is for . A control plane equivalent metric doesn't exist.

One option is to autoscale on a new ate.workerpool.demand (filtered on pool_exhausted blocked reason) metric that counts actors that want a slot:

attribute	values
`ate.workerpool.name`	pool name; empty when nothing in the fleet was eligible
`ate.sandbox.class`	`gvisor` / `microvm`
`ate.demand.state`	`resident` (holding a slot now) / `blocked` (rejected/still retrying)
`ate.blocked.reason`	`none`, `pool_exhausted`, `node_locality`, `no_eligible_pool`, `unknown`

/assign

Shrutiya Mohan (@shrutiyam-glitch)

Good timing, this overlaps with the metrics work I started in #433.

FYI #514 adds ate.scheduler.assignment.duration with an outcome label, so the no_free_worker rate gives you a view of this pressure. Not something you'd HPA on directly, but probably a good thing to cross reference and document.

I have an ate.workerpool.queue.depth gauge wip working locally (ateapi observable gauge with a pool label) that was waiting on
#221.

Couple of thoughts:

The resident state mostly duplicates ate.workerpool.workers{state="assigned"}, so the new signal is the blocked count. A cheaper version could be a gauge that only counts waiting actors, with the blocked reason as a label. The HPA lgic also gets simpler if you can scale on the gauge directly instead of filtering one state out of a combined one.
Naming: we've been reserving top-level ate.* namespaces for cross-cutting concepts, so ate.demand.state / ate.blocked.reason should probably live under the owning domain instead (see the ateattr registry already has ate.workerpool.name and ate.sandbox.class). No strong feelings on demand vs. queue.depth for the metric name itself.

Shrutiya Mohan (@shrutiyam-glitch) since you are assigned to this, happy to help if needed, I can take the ateapi gauge as the next part of the platform
metrics work, or just review it.

Krisztian F (@krisztianfekete)

Thanks for sharing your thoughts Krisztian F (@krisztianfekete)!

FYI #514 adds ate.scheduler.assignment.duration with an outcome label, so the no_free_worker rate gives you a view of this pressure. Not something you'd HPA on directly, but probably a good thing to cross reference and document.

From what I'm seeing in the PR, this metric also has a worker pool label. It would be interesting to experiment autoscaling on a percentile of this metric too (e.g., using HPA to try to keep the P90 of assignment duration < T ms).

I have an ate.workerpool.queue.depth gauge wip working locally (ateapi observable gauge with a pool label) that was waiting on #221

Is there any difference between the ate.workerpool.queue.depth metric and ate.workerpool.demand{state=blocked}? I guess it's only the blocked.reason label that's missing in the former? Is this something we could add to ate.workerpool.queue.depth too? So we can dedupe the metric and also run some experiments to see the autoscaling responsiveness.

Shrutiya Mohan (@shrutiyam-glitch) also has ideas for other metrics that we could use for autoscaling, so we can brainstorm more here

ate.workerpool.workers counts workers by state, so it's clamped by capacity: eg, when a pool of 5 is fully assigned it reports 5, whether one actor is waiting or a thousand are.

I think this will be maxed out at --parked-request-max, so we cannot really catch spikes?

Is there any difference between the ate.workerpool.queue.depth metric and ate.workerpool.demand{state=blocked}? I guess it's only the blocked.reason label that's missing in the former? Is this something we could add to ate.workerpool.queue.depth too? So we can dedupe the metric and also run some experiments to see the autoscaling responsiveness.

The park budget is 5s, so with HPA syncing every 15s we could easily sample at the wrong timestamp. The budget_exhausted park outcome is probably the better signal.
See:

substrate/cmd/atenet/internal/router/parking.go

Lines 28 to 44 in aa9b7b8

    
           const ( 
        
           	defaultParkedRequestBudget = 5 * time.Second 
        
           	// defaultParkedRequestMax is sized together with the ext_proc cluster's 
        
           	// circuit breaker (--extproc-max-requests, derived as twice the lot by default): each parked 
        
           	// request holds one ext_proc stream, i.e. one active request against that 
        
           	// cluster, for its entire wait. Startup validation keeps an explicit breaker >= the 
        
           	// lot, and the default pair (1024 lot / 2048 breaker) leaves equal headroom 
        
           	// for the fast path. See buildCluster in xds.go. 
        
           	defaultParkedRequestMax = 1024 
        
           	// Retry cadence between resume attempts while a request is parked: a gentle 
        
           	// exponential backoff. 
        
           	defaultParkedRequestRetryInterval = 100 * time.Millisecond 
        
           	defaultParkedRequestRetryFactor   = 1.1 
        
           	defaultParkedRequestRetryJitter   = 0.1 
        
           )

shrutiyam-glitch also has ideas for other metrics that we could use for autoscaling, so we can brainstorm more here

Scaling on blocked count happens after users are already waiting, which is not great given the warmup time. HPA targets can't be zero, so we'd also have some blocked actors stuck in waiting. Thinking about making idle headroom, e.g. workers{state="idle"} the primary signal + using burst, which is closer to the warm buffer idea in #198. Wdyt?

Navigation Menu

Repository navigation

Identify what metrics we should use for HPA-based workerpool autoscaling #671

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	config.yaml: \|
	externalRules:
	- seriesQuery: '{__name__="ate_workerpool_workers"}'
	resources:
	namespaced: false
	name:
	matches: "^(.*)$"
	as: "$1"
	# max() across ate-api-server replicas since each replica reports the
	# global count.
	metricsQuery: 'max(<<.Series>>{<<.LabelMatchers>>}) by (ate_workerpool_namespace, ate_workerpool_name, ate_worker_state)'

	const (
	defaultParkedRequestBudget = 5 * time.Second

	// defaultParkedRequestMax is sized together with the ext_proc cluster's
	// circuit breaker (--extproc-max-requests, derived as twice the lot by default): each parked
	// request holds one ext_proc stream, i.e. one active request against that
	// cluster, for its entire wait. Startup validation keeps an explicit breaker >= the
	// lot, and the default pair (1024 lot / 2048 breaker) leaves equal headroom
	// for the fast path. See buildCluster in xds.go.
	defaultParkedRequestMax = 1024

	// Retry cadence between resume attempts while a request is parked: a gentle
	// exponential backoff.
	defaultParkedRequestRetryInterval = 100 * time.Millisecond
	defaultParkedRequestRetryFactor = 1.1
	defaultParkedRequestRetryJitter = 0.1
	)

Identify what metrics we should use for HPA-based workerpool autoscaling #671

Description

Activity

shrutiyam-glitch commented on Jul 31, 2026

krisztianfekete commented on Aug 4, 2026

laoj2 commented on Aug 4, 2026

krisztianfekete commented on Aug 6, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions