Repository navigation
Guided Application Apply recovery: reconcile cloud resources, review State repair and resume the same order #157
Description
Current status (2026-10-25)
Guided recovery is deployed in Dev, including original Application Owner self-service under a valid execution delegation, the complete orderable Application resource catalogue and separately approved targeted revocation of credentials whose secret was lost. PRs #159 and #160 are merged. The original hosted fixture completed recovery, a fresh Plan and a separately user-started Apply through the supported UI. The observations below describe the preserved historical failure, not its current state. Remaining crash/publication ambiguity acceptance keeps this issue open; see the latest deployment evidence comment.
Problem
The disposable hosted Apply hard-loss drill in #95 now supplies a real acceptance fixture: a stopped Runner, an immutable interrupted-Apply receipt, a partial persisted State, its orphan lock and matching cloud resources missing from that State. The current UI correctly blocks unsafe replanning but does not provide a complete supported recovery path.
The user explicitly wants to preserve this fixture and qualify the product workflow against it. A workstation-only repair, exported operator credentials or agent access to private scripts is not an acceptable product solution. The previously proposed manual unlock/remote State publication was not approved and has not been executed.
Observed evidence
- The user started the disposable Apply via the normal authenticated UI; the exact Runner was stopped after a stable native partial S3 checkpoint.
- Independent CF inspection confirms STOPPED/DOWN. The owner used Check interrupted Apply; independent DB inspection confirms terminal
reconciliation_required/cancelledwith its immutable receipt. - The remote checkpoint retains 3 managed resources, serial 1, with the original orphan lock.
- Read-only cloud reconciliation identifies the matching Application network and automation service account outside the checkpoint. The intended project-role assignment also exists. No automation key exists. The storage credentials group is the platform-created default and must not be adopted as an Application-created group.
- Native imports into a private local copy, using the original bound inputs/source and pinned OpenTofu 1.12.6 / provider 0.114.0, preserve the original resources and lineage. The candidate has 6 managed resources, serial 4. A local diagnostic preview retains 6 and proposes 6 creates, with no updates, deletes or replacements. It is not a saved Configurator Plan eligible for Apply.
- Remote State and lock remain unchanged. No manual completion, database execution bypass, duplicate order or additional infrastructure Apply is authorized by this issue.
This does not establish an OpenTofu bug: cloud operations accepted/in flight at hard loss can also explain resources outside the checkpoint. Keep #140's local-backend finding distinct unless the cause is independently demonstrated.
Product workflow to implement
- Assess: an explicitly started, server-side recovery assessment verifies the exact order/owner/tenant, original source and backend, old Runner termination and closed/expired execution capabilities. Read State/lock and the supported cloud inventory using the product's authorized credential mechanisms. No local CLI login or credential download is required for the user.
- Compare: show retained resources, unambiguous existing resources absent from State, missing resources and unresolved conflicts. Use exact project/region/address/identity evidence; names alone never authorize adoption. Missing/expired access, incomplete pagination, unsupported resource types and ambiguous matches must remain explicit blockers.
- Propose: build an encrypted, versioned recovery candidate with native supported imports against a copy of the existing State and the original source/inputs. Preserve lineage and every original resource identity. Never infer a lost secret from metadata; keys whose private material is unavailable need a separate explicit rotation/replacement decision. No cloud create/update/delete occurs in assessment or import preparation.
- Review and approve: present concrete proposed State additions and a diagnostic preview. Bind approval to tenant/order/job/source, exact cloud identities, State version/hash and lock identity. Keep execution approvals separate. The current conservative Platform Engineer authorization for ambiguous completion must not be silently widened; make the requester/reviewer handoff visible in the product.
- Commit safely: revalidate authorization, stopped Runner, expired old capabilities, cloud evidence, original State and exact orphan lock. Archive protected original State/lock evidence. Release only the verified orphan lock and publish the reviewed candidate under native State locking, lineage/serial guards and durable intent/outcome receipts. An uncertain write/response requires reconciliation, never blind replay or
state push -force. Unrelated locks, newer State or changed resources invalidate the proposal. - Resume the same order: after independently verified State publication and free lock, record the existing guarded recovery clearance. Allow a fresh normal saved Plan of the same instance/backend. A new infrastructure Apply still needs its usual separate review and user approval; never replay the old saved Plan.
Acceptance
- Finish this exact preserved disposable fixture via the normal product UI and service/Runner infrastructure, without workstation-only State repair, arbitrary SQL writes, session impersonation or a replacement order.
- A fresh hosted Plan retains all six identified existing resource identities, creates only the missing six resources and proposes no unintended update, delete or replacement; re-evaluate actual counts if the independently observed cloud state changes.
- A separately reviewed, user-started fresh Apply succeeds and independent State/cloud checks confirm retained identities.
- Exercise tenant/owner/reviewer authorization, expired or revoked credentials, active/unknown/restarted Runner, foreign lock, changed State/object version, stale approval, ambiguous inventory and missing pagination pages.
- Exercise crash/timeout/response loss before and after unlock/publication, repeated clicks and concurrent recovery attempts; no double commit, overwritten newer State or stuck new lock is accepted.
- Store durable immutable recovery receipts and protected backups/candidates; do not expose raw State, secrets or private inventories through public logs, chat or issue artifacts.
- Provide clear German/English user guidance and precise UI next actions. Distinguish regular reported failure, hard Runner loss, incomplete State, ambiguous cloud state and a completed recovery.
Related: #95 operational acceptance, #140 first-Apply intermediate checkpoint investigation, #155 interrupted-Apply diagnosis and #156 hosted observer tooling. Diagnosis is shipped; the full guided State/cloud recovery above is not implemented yet. Keep the live fixture intact until the product path is ready and concrete mutation approval is obtained.
Activity
lweberru commentedon Oct 25, 2026
Implementation is in PR #158. The guided path separates a read-only assessment, a private native-import candidate and diagnostic preview, explicit original-Platform-Owner approval, once-only native lock/State publication, and independent remote verification before same-order replanning. Unknown results are inspected without replay. New jobs and separate clearance paths are serialized against an active recovery.
Local evidence: 1,195 application tests passed, PostgreSQL integration 70/70, platform database 63/63, browsers 346/346, and complete Runner packaging/native contracts passed. Actual STACKIT token authentication and inventory reads succeeded on the preserved disposable fixture. Two real preparations by the packaged native worker produced byte-identical candidates: three imports, all three original instances retained, preview six creates/six no-ops and zero updates/deletes/replacements. Only the CF provider-mirror path was adapted for local execution. The initial encoded-
@key-list 403 was reproduced, corrected and covered by a regression test.Remote fixture State and lock remain unchanged. Hosted publication requires a separate concrete UI approval after deployment; it has not been performed. Keep this issue open for that evidence and catalogue limits. The initial safe adoption catalogue covers the retained Public project/local network/automation Service Account/explicit project role assignments. Corporate routing, custom roles, orphan service instances and lost private key material remain explicit blockers; #140 is separate.
feat(configurator): add guided Application State recovery (#158)
lweberru commentedon Oct 25, 2026
PR #158 is merged. Dev release 40195876227 completed successfully from
49a57c8602972fdcdb9ccc43e281013ff66fa0de, usingdev-skip-assistant-qualificationand runner pool size 1. Assistant remains enabled.Independent runtime checks confirm the actual new app/droplet source, aligned Runner source, installed recovery migration, one ready/live warm Runner, protected recovery routes and the new UI. A short-lived route-free diagnostic app used only a READ-ONLY DB transaction and was removed. The exact interrupted Apply and its immutable interruption receipt are preserved; there are zero recovery requests and zero repair write intents so far.
A direct double S3 read also confirms the original State checksum, serial 1, three managed instances and the exact original orphan lock are unchanged. No native unlock, State publication or infrastructure Apply has been performed. Next acceptance is the real read-only UI assessment by the original Platform Owner, then review of its concrete proposal before any separately confirmed publication. Keep this issue open; hosted publication is not yet qualified.
lweberru commentedon Oct 25, 2026
The preserved disposable fixture has now completed the guided hosted UI workflow and resumed successfully in the same order.
This qualifies the end-to-end recovery and successful resumption of this supported Public/local-network fixture without workstation State repair, replacement order or replay of the interrupted saved Plan. The final State verification is independent; the Plan/Apply execution output was supplied by the user through the normal UI.
Keep this issue open for the remaining catalogue and failure-mode acceptance scope. This result does not establish recovery support for Corporate routing, custom roles, orphan service instances or lost private key material, nor does it complete #140. Previously documented unsupported/ambiguous cases remain blockers.
lweberru commentedon Oct 25, 2026
Follow-up implementation is in PR #159, extending the catalogue to the resource types reachable from the pinned Application root rather than requiring a real hard-stop drill for each type. CI guards compare native types/addresses with the recovery strategy catalogue; a deterministic matrix covers all 8,192 retained/missing subsets of the thirteen importable instances in the full Corporate fixture.
The original Application Owner can assess, explicitly approve and inspect recovery of their own order under the still-valid original execution delegation and verified matching identity. The original Platform Owner retains support access. This does not widen the separate manual operator-attestation API or expose platform credential profiles.
Lost automation/S3 secrets have an explicitly reviewed, separately confirmed revocation manifest. Once-only durable intents precede targeted fixed-endpoint revocations; only independently verified absence permits unlock/publication. Incomplete ID-only credential State entries may be removed through the same reviewed manifest using native local State removal; fully saved credentials and other identities are protected. Replacement credentials require a fresh ordinary reviewed Plan/Apply. Unknown deletions/publications are not blindly replayed.
Local validation: full application check 1,215 passed (74 gated/skipped), real PostgreSQL integration 70/70 with owner-issued recovery tasks and support access, platform database 63/63, full browsers 348/348 and final recovery scenarios 10/10, complete Runner package/native contracts and final native/mirror regressions passed. The existing successful hosted fixture remains intact; these tests did not create another customer order, mutate its State or revoke a real credential. The expanded release is pending current-head CI and Dev deployment; this is not a claim that every type has undergone a new live interruption.
feat(recovery): cover Application resources and owner self-service (#159
fix(recovery): preserve intervening State during publication (#160)
lweberru commentedon Oct 25, 2026
Dev deployment and independent operational verification completed successfully.
d08b832479f0b9ef67aa041d0229f25f448fd47d.dev-skip-assistant-qualification,runner_pool_size=1,build_only=false. Assistant remains enabled.The issue remains open for the remaining crash/timeout/publication-ambiguity acceptance and hardening. Catalogue coverage and owner self-service are shipped; arbitrary ambiguous recovery outcomes are not declared solved.