Skip to content

fix(worker): continue an interrupted job instead of restarting it #175

Open

Description

@valentinalexeev

🤖 AI agent: refs #168 and #174 (a job that is killed must continue, not restart).

Filed from the owner account at the owner's request in chat; the bot identity was not available in this session.

Problem

After a killed pod restarts, the loop clears the job locks (agent-loop.sh:230) and re-dispatches the issue or PR, but the job starts over:

  • worktree() resets the branch to origin/<base> with reset --hard and clean -fd: uncommitted work and Claude's local commits are lost.
  • Claude runs again without --resume: the tokens of the first run are spent twice.
  • issue_job posts a second "picked this up" comment.
  • Probably stuck (not tested): if the kill came after the push and before the PR was created, the re-run starts from origin/<base> again and the forge sidecar refuses the non-fast-forward push (images/forge/git.go:278), so the job fails and retries with backoff without ever opening the PR.
  • Feedback jobs write the cursor seen/pr-N only at the end: a kill after the push or reply and before the cursor write gives a duplicate reply on the PR.
  • An open approval request (feat: agent permission requests and approvals in the management UI #67) of the killed job is not voided.

Change

  1. Continue from the branch. If the job branch exists on the remote or has local commits (including the work-in-progress commit of the SIGTERM handler, feat(operator): drain agents before replacing their pods on upgrade #174), the worktree starts from it, not from origin/<base>.
  2. Resume the Claude session. Keep the session id per job key on the state volume and run claude --resume <id> with a short "you were interrupted, continue" prompt; fall back to a fresh run if the session is gone.
  3. Idempotent steps. Before creating a PR, look up an open PR for the head branch and use it. Skip the "picked up" comment on a re-run (marker on the state volume). Write the feedback cursor right after the reply is posted, and record the posted comment id so a re-run does not post it again.
  4. Approvals. On start, void open approval requests of jobs that are no longer running.
  5. Tests in images/worker/test for each case above; docs (docs/concepts.md agent state, docs/troubleshooting.md) and CHANGELOG.

Acceptance criterion

Worker tests cover: kill during the Claude run (resumes the session, keeps local commits), kill after push before PR (opens the PR, no new run), kill after a feedback reply before the cursor write (no duplicate reply). On a live instance, deleting an agent pod in the middle of an issue job ends in one PR, one "picked up" comment, and Claude usage that is not doubled.


Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

AICreated by an AI agentbugSomething isn't workingworkerAgent loop and worker image

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions