Repository navigation
Add Helm charts for installing Agent Substrate #79
Description
Agent Substrate is currently installed through hack/install- ate.sh / hack/install-ate-kind.sh, which apply raw
manifests, run ko, generate prerequisite secrets, and handle
Kind/GKE-specific differences.
We should add Helm-based installation with two charts:
agent-substrate-crds: installs only Agent Substrate CRDs.agent-substrate: installs the runtime/control-plane
system.
Splitting CRDs into their own chart follows current Helm
operational best practice: CRDs are cluster-scoped, upgrade-
sensitive, and often managed by cluster/platform admins
separately from namespaced workloads.
Proposed Chart Layout
charts/
agent-substrate-crds/
Chart.yaml
crds/
ate.dev_actortemplates.yaml
ate.dev_workerpools.yaml
README.md
agent-substrate/
Chart.yaml
values.yaml
templates/
README.md
Motivation
A Helm-based install would make it easier to:
- Install Agent Substrate without relying on repo-local shell
scripts. - Support GitOps tooling such as Argo CD and Flux.
- Manage CRDs independently from runtime workloads.
- Configure images, ValKey, TLS, storage, and observability
through values.yaml. - Provide clearer install, upgrade, and uninstall workflows.
Scope
agent-substrate-crds
This chart should install:
- WorkerPool CRD
- ActorTemplate CRD
- Any future ate.dev CRDs
Acceptance criteria:
- helm install agent-substrate-crds ./charts/agent-substrate-
crds - CRDs are installed using Helm’s crds/ convention.
- Chart README documents CRD upgrade behavior and uninstall
caveats. - Runtime chart does not template or own CRDs directly.
agent-substrate
This chart should install the core system components:
- ate-api-server
- ate-controller
- atelet
- atenet-router
- atenet-dns
- ValKey, optionally
- podcertificate-controller
- RBAC, ServiceAccounts, Services, ConfigMaps, Secrets, and
projected volumes
The runtime chart should assume CRDs are already installed and
document installing agent-substrate-crds first.
Configuration Requirements
Expose values for:
- Component images, tags, pull policies.
- Internal vs external ValKey.
- Redis/ValKey TLS and auth settings.
- GCS/S3/local storage backend settings.
- Kind/local development profile.
- GKE/cloud profile.
- OTLP and Prometheus configuration.
- Existing secret names for session JWT/CA pools and pod
certificate CAs.
Bootstrap Secret Handling
The current install script generates prerequisite secrets such
as:
- session-id-jwt-pool
- session-id-ca-pool
- service-dns-ca-pool
- pod-identity-ca-pool
- valkey-ca-certs
- ate-api-server-envvars
The chart should document whether these are:
- Provided by the user before install.
- Generated by Helm hooks or bootstrap Jobs.
- Supported through both modes.
Externally supplied secrets should likely be the production/
default path.
Acceptance Criteria
- helm install agent-substrate-crds ./charts/agent-substrate-
crds installs CRDs. - helm install agent-substrate ./charts/agent-substrate -n
ate-system --create-namespace installs the core system after
CRDs exist. - helm uninstall agent-substrate -n ate-system removes runtime
resources cleanly without deleting CRDs. - Runtime chart fails clearly or documents behavior when CRDs
are missing. - values.yaml covers current hack/install-ate.sh
configuration. - Kind/local and cloud/GKE install paths are documented.
- CI validates helm lint and helm template for both charts.
- Existing script-based install either keeps working or is
updated to render/apply the Helm charts.
Non-Goals
- Packaging demo apps into the core runtime chart.
- Solving production-grade certificate/key rotation in the
first version. - Removing the existing development scripts immediately.
Implementation Notes
Current install logic lives in:
- hack/install-ate.sh
- hack/install-ate-kind.sh
- manifests/ate-install/
- manifests/ate-install/generated/
- manifests/ate-install/kind/
The current manifests use ko:// image references. The Helm
charts should decide whether they require pre-published images
or whether development docs continue using ko to publish
images before Helm install.
Activity
BenTheElder commentedon May 27, 2026
I don't think we should make helm a requirement. Not everyone uses it, and it's a pretty heavy dependency.
I strongly encourage structure-aware yaml manipulation (e.g. kustomize) over text templating.
I also think the ko based workflows have been very helpful for rapid iteration and I don't think we want to lose that.
I think it may be slightly premature to start publishing builds. We don't have defined versions and releases yet.
We will need to do so eventually, but see:
https://github.com/agent-substrate/substrate/blob/9bd41df2a36d4779206babbad8c4b6bf1f6e0884/README.md#status-and-compatibility
a4-a4s1 commentedon May 27, 2026
Stepping back from the helm-vs-kustomize debate: both produce manifests that a GitOps tool can sync. The harder constraint may sit elsewhere.
Two needs #79 is folding together:
Both needs trace back to the same thing: when stable image references become available. The comment above flagging "premature to publish builds" + the README status-and-compatibility note both point there.
One Q for maintainers: is there a target where stable builds become the goal? That timing probably anchors the packaging shape more than helm-vs-kustomize directly.
[🤖a4s1]
EItanya commentedon May 27, 2026
Just to be clear, I’m mostly focused on reproducible installs via popular tooling. From my experience helm is BY FAR the most popular and widely used kubernetes installation mechanism, usually via Argo and other gitops tools at the org level. My main goal in opening this issue is starting the discussion on how users will install/manage installations of substrate.
I saw #64, but my experience is that GitOps is the preferred method most of the time, so I wanted to get that discussion going here. Happy to modify the issue to be more about reproducible gitops friendly installs if that would help.
a4-a4s1 commentedon May 27, 2026
+1 on broadening the issue toward "reproducible GitOps-friendly installs" — that reframe sidesteps the tooling-comparison and surfaces the actual question.
A few observations on how that broader scope intersects with what's already in-tree:
ko-rapid-iteration and pre-published images aren't mutually exclusive. Dev loop keeps
ko publishfor the tight feedback path; release path publishes versioned images that the GitOps install consumes. The two paths don't share a build but can share a manifest shape (kustomize base + overlay), so the "ko vs published" Q collapses into "two consumers of the same manifests."The structural Q the broader scope surfaces: where does
manifests/ate-install/sit relative to the GitOps install path? Today'smanifests/isko resolveoutput; a published-image install would either (a) re-render the same base with image refs swapped in, or (b) ship a separate rendered tree. (a) keeps one source of truth; (b) decouples release cadence from manifest changes.ate-operator: Make it easy to install and manage (multiple) substrates #64 (mtaufen ate-operator install) and this issue overlap on the install-time concerns but differ on the operator-vs-charts axis. Worth a forward-link between the two once the scope here settles, so the operator-install path has a place to land.
If the issue gets retitled, would it be useful to enumerate the install-time decisions explicitly (release artifact format, version pinning, RuntimeClass parameterization, multi-tenant overlay shape) so the conversation can converge on each in turn?
[🤖a4s1]
thockin commentedon May 31, 2026
Given how very in-flux things are -- how important is this RIGHT NOW?
What I want to avoid is having the same set of steps encoded into 2, 3, 4 different places. If we adopt helm, we should EOL the scripts or make them thin wrappers. Building images and things is still needed, so we can't totally get away from scripts?
EItanya commentedon Jun 1, 2026
I think maybe my wording of this issue was a bit too prescriptive. I think my main goal here is to make it easier for people to try substrate. I totally understand that things are in flux, and we don't want to commit to any specific shape, but requiring cloning and building is a pretty large hurdle. Maybe we can start with publishing images and a manifest to point folks too. Or is the fear that everything is too early for that?
thockin commentedon Jun 1, 2026
In this regard, my nature is very conservative. I am terrified of the idea that people will try it, and realize how premature it is, and then never come back. "You only get one shot at a first impression"...
BenTheElder commentedon Jun 1, 2026
Exactly. I'm concerned that offering "install this in your production v1.31 cluster via one copy-paste command" will invite end-users to depend on a very early pre-alpha before it is ready for that, and for those users it will be a bad first impression.
For the sort of cloud native developer that wants to evaluate an early POC and possibly contribute, the barrier should be pretty low, we've specifically invested in making the local development flow have minimal requirements and it's quite fast.
We do have room to make it even lower if needed (e.g. we could obtain go and kubectl for you if that's really a barrier, I think for the current audience it's probably not).
We also assume a lot of interested users are using AI, which can certainly figure out cloning and running the demo scripts. We've tested that of course.
I don't think that hurdle is the wrong level at this "pre-alpha" stage.
I do think eventually we need to have pre-built binaries, but that should probably be when there's some commitment to compatibility.
At that point, I'd like to avoid requiring helm, but I don't think it's unreasonable to offer it.
A number of Kubernetes projects do this by having pre-rendered manifests available to patch with your favorite tool alongside a published chart.
The use of the beta cert APIs is similar, Taahir Ahmed (@ahmedtd) expects we can GA those in k8s alongside GA-ing substrate so it makes sense to continue driving in that direction even though they're an obstacle for older k8s releases.
bnaylor commentedon Oct 14, 2026
Claude (tigerteam-planner session) for bnaylor (@bnaylor):
A data point for this thread from kube-agents (gke-labs/kube-agents#2935), where a platform agent was asked to "create a new cluster for me that has Agent Substrate installed on it."
We route infrastructure changes through a pull request against the customer's GitOps repo, so nothing installs until a human merges it. For Substrate the cluster half works fine as declarative config: Standard mode, non-E2 machines, the two beta APIs at creation, the snapshot bucket and its Workload Identity grants, the node pool label. The workload half doesn't, because the only install path is
install.sh->ate-setup deploy, which rendersmanifests/ate-installand fills placeholders (${SUBSTRATE_VERSION_SUFFIX}inatelet.yaml, for one). So today our agent proposes the cluster and then stops, saying there's no declarative package to point at.Totally understand the "not ready for one-command installs" concern above, and we're not asking for a Helm chart. Something as small as EItanya's suggestion would cover it: a rendered, versioned manifest bundle attached to each release (or the parameters
ate-setupsubstitutes, documented), so Config Sync / Flux / Argo can sync it and pin a version. It could stay clearly labelled pre-alpha.Not urgent on our side. If that's not something you want to publish yet, that's a fine answer and we'll keep the stop.
BenTheElder commentedon Oct 14, 2026
I think this is something we're strongly considering for GA, but we have SO MANY other things in flight right now that things like ... figuring out upstream binaries (we have made progress on release automation but it's not all there yet) will need to come first before revisiting this.
I agree having a rendered manifest is something we want.
Merge pull request agent-substrate#79 from kagent-dev/sync/upstream-m…
build(ci): build-images workflow from upstream release tags, official…