api: opt-in single-attempt failure policy for Application HelmReleases, with a terminal status #5109

Problem / motivation

Managed applications created through apps.cozystack.io have no terminal failure. The HelmRelease that cozystack-api generates for every Application uses RetryOnFailure with a 30s RetryInterval and no Remediation (pkg/registry/apps/application/rest.go, convertApplicationToHelmRelease), so a failed install or upgrade is retried for as long as the Application exists. The general problem is tracked in #4155; this issue proposes one concrete, opt-in option from it.

The case that needs it: an external orchestrator, such as a customer-facing panel on a production v1.6.x deployment, creates databases (Postgres, MariaDB, ClickHouse, MongoDB) through the API and has to give its users a final answer when creation fails. It needs to mark the instance as failed, be sure it will not come up later on its own, and let the user create a replacement. Today it cannot tell "still retrying" from "will never succeed": convertHelmReleaseToApplication copies only the Ready and Released conditions, and Ready keeps flipping between False/InstallFailed and Unknown/Progressing on every retry. The only way to stop the retries is to delete the Application.

Proposed solution

1. A new per-kind ApplicationDefinition annotation, alongside the existing ones in pkg/config/config.go (release.cozystack.io/helm-install-timeout, release.cozystack.io/helm-upgrade-timeout, release.cozystack.io/helm-install-disable-wait), for example release.cozystack.io/helm-failure-strategy: stop.
2. When a kind sets it, convertApplicationToHelmRelease renders Strategy: RemediateOnFailure with Remediation{Retries: 0} for both install and upgrade, leaves RemediateLastFailure unset and does not set RetryInterval (the Flux CRD rejects retryInterval with RemediateOnFailure).
3. Per the Flux docs, such a release makes one attempt, performs no uninstall or rollback (remediateLastFailure defaults to false when retries is 0), and stalls with Stalled=True, reason RetriesExceeded. The failure counters reset on the next values or chart change, so updating the Application retries it.
4. convertHelmReleaseToApplication also copies the Stalled condition, so API consumers can tell a terminal failure from a retry.
5. Kinds that do not set the annotation keep today's behaviour unchanged, including the 30s retry decoupling discussed in generated HelmReleases retry a failed install forever, with no cap, backoff, or notion of a terminal failure #4155.

Requested as a backport to the 1.6 line (1.6.6), since the deployments that need it run 1.6.x.

Scope check

• This affects a single component / package (if it spans many, consider a design proposal)
• I can describe the expected behavior concretely (if not, consider a Discussion first)

Alternatives considered

• Make the orchestrator decide alone (a timeout or a threshold on status.failures, then delete the Application). It works today, but the platform keeps retrying until the delete lands, the orchestrator needs HelmRelease read access to see the failure counter, and nothing in the Application API reports the decision.
• RemediateOnFailure with Retries > 0. Rejected for databases: between attempts it runs an uninstall for installs, which triggers the post-delete PVC cleanup hooks, and a rollback for upgrades, which strips the spec.plugins the CNPG backup driver adds and stops WAL archiving.
• Changing the platform-wide default. Rejected: a slow volume or image pull past the install timeout would become terminal for every kind, which is the convergence generated HelmReleases retry a failed install forever, with no cap, backoff, or notion of a terminal failure #4155 wants to keep.

Additional context

• generated HelmReleases retry a failed install forever, with no cap, backoff, or notion of a terminal failure #4155: generated HelmReleases retry a failed install forever (general problem and options).
• Generated HelmReleases set RetryOnFailure with a nil Remediation, which can refuse to reinstall an absent release #3554: nil Remediation interacts with RetriesExhausted on an absent release; any change to the remediation block should be checked against it.
• Flux HelmRelease docs on install/upgrade strategies, remediation.retries, remediateLastFailure and the Stalled condition: https://fluxcd.io/flux/components/helm/helmreleases/