Repository navigation
Add pre-upgrade safety checks and reusable policy evaluation #2941
Description
Current situation
- Today, operators upgrade components one at a time using the upgrade runbook.
ate-setupdoes not check whether the version change is supported or whether earlier upgrade steps have finished before starting the next step.
An unsupported version or an update applied too early can leave components unable to work together, interrupting API calls or actor operations.
This issue proposes adding read-only checks after GA to catch these mistakes before the next update. Operators would still run the upgrade manually.
Failure cases and proposed checks
The following cases map the runbook and version-skew policy to checks that ate-setup could perform:
| Failure case | Proposed check |
|---|---|
| Unsupported version: The selected upgrade or rollback is not supported. | Check the source and target releases against the policy. For rollback, establish which release the upgrade started from. |
| Wrong cluster or unclear state: The selected Kubernetes context differs from the expected context, the saved record differs from running versions, or versions cannot be established. | Compare the selected and expected contexts. Report verifiable differences between recorded and running versions. Return unknown when evidence is missing. |
| Update started too early or with the wrong worker image: Earlier updates are unfinished, a worker rollout is paused before the control-plane step, or the proposed worker image is incorrect. | Check prerequisites for the requested step, including rollout completion and worker pause status. When a proposed worker image is supplied, check its sandbox type and digest. |
Proposed implementation
Slice 1: Policy evaluation
Add a Go package that validates upgrade and rollback transitions against the version-skew policy and returns reasons for rejected transitions.
Keep this logic separate from Cobra and deployment execution so other installation tools can reuse it.
Slice 2: Read-only ate-setup check
Add a command such as:
ate-setup upgrade check --from-config old.yaml --config new.yaml
The command would run the checks above and return ready, blocked, or unknown. It should identify the next manual step when the available evidence supports that conclusion.
Checks for a particular step or worker image need explicit inputs. The example above illustrates the entry point, not the final flag design.
Slice 3: Output for other installation tools
Provide human-readable and machine-readable results containing the observed state, decision, reasons, and missing evidence. Future Kustomize, Helm, provider, and CI workflows could consume these results without depending on ate-setup deploy.
Image digests and locally built images may not identify a release reliably. Until a reliable way to identify live component releases is agreed, report unknown. Additional workload metadata can be considered separately.
Acceptance criteria
- Unit tests cover the documented release-transition rules.
- Cluster-state tests cover the failure cases above, including missing evidence and partial upgrades.
- The checker makes no cluster changes.
- Human-readable and machine-readable output report the same decision.
- The runbook explains when and how to run the checks.
Out of scope
- Automatically executing the upgrade or rollback
- Creating database backups
- Maintaining a local upgrade-progress file
- Preventing updates when the checker is bypassed
- Selecting the permanent post-GA installer
Activity
HavenXia commentedon Oct 21, 2026
Thanks, this matches what we discussed.
In "Failure cases and proposed checks":
This is already handled,
deploy apiserveralready refuses the bundled database over an external one.I suggest we start small: a version check from
old.yamlandnew.yaml, as a function insideate-setup, and leave the cluster-state checks and machine-readable output (helm and others) until something needs them.HavenXia commentedon Oct 21, 2026
One update: Steven Shriver (@stshrive) is drafting an installer redesign where every install path, including Helm and kustomize, runs an in-cluster
ate-setupbootstrap Job first.The version check could run there, so it covers every path and can't be skipped. We can discuss more once that get finalized.
tryambak2019 commentedon Oct 21, 2026
Thank you very much, Haven Xia (@HavenXia) !
I removed the database case from the issue description.
Based on your feedback, I propose the following scope for the first milestone.
Policy evaluation
Acceptance criteria
This Go function can be implemented independently of Steven’s installer redesign. Its invocation point, including possible bootstrap Job integration, can be decided after redesign is finalized.
Excluded for current implementation:
Please confirm that this captures the intended scope
fiunchinho commentedon Oct 29, 2026
One more case for the policy evaluation: refusing an upgrade whose source release is older than the oldest release the target can migrate from. Today an operator learns that from a failed ate-api start (or a silently unconverted row); with a check in the bootstrap Job it fails before any component moves. Happy to help with that case once the first milestone's shape is settled.
HavenXia commentedon Oct 29, 2026
tk (@tryambak2019) sorry for the slow reply, that scope works.
The runbook copies the record into
old.yamlandnew.yamlbefore step 1, socluster.substrateVersionin the--configfile is the release the upgrade started from. ParseFile skips that section today, so the check has to read it.It doesn't need to be in 1.0.0: the upgrade runs the new release's ate-setup, so the first skipped minor it can catch is 1.0 → 1.2.
Jose Armesto (@fiunchinho) I think that's the skipped-minor rule: we have
doc/version-skew-policy.mdsupporting one minor per upgrade because the schema evolution assumes it, so the previous minor is the oldest release a target accepts.fiunchinho commentedon Oct 29, 2026
Makes sense 👍