Skip to content

Horizontal autoscaling for ate-apiserver #3706

Description

The ate-apiserver today is statically sized. We default to two replicas. This task is to explore how to use HPA to automatically scale it based on demand. The ate-apiserver is designed to scale horizontally as any replica can theoretically handle any request (no sharding or affinity is needed).

This task is to explore what metrics are the best candidates for scaling (cpu utilization, pgx connection pool utilization, etc), how to properly tune HPA targets, and how to plumb everything together.

cc Luiz Oliveira (@laoj2) Scott Funkenhauser (@sfunkenhauser)

Activity

  1. laoj2 commented on Nov 6, 2026

    @laoj2
    Collaborator

    Learnings from https://docs.cloud.google.com/kubernetes-engine/docs/how-to/design-substrate-infrastructure#size-system-infra-components:

    • Horizontal scaling stops improving throughput after a shared dependency (such as PostgreSQL CPU or connection limits) reaches capacity. Size your system infrastructure components and database together.
    • Each postgres replica opens up to poolMaxConns + 6 connections (owner 2, watch 3, readiness 1, in cmd/ateapi/internal/store/atepg/atepg.go). The sum across replicas must stay under max_connections, so the HPA maxReplicas should be floor((max_connections - reserved) / (poolMaxConns + 6)).
    • minReplicas >= 2, to keep today's availability
    • Candidate metrics: CPU utilization and RPC rate per pod.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/apiUser-facing API changesarea/api-machinerykind/designDesign discussion, investigation, or research required before implementation

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions