Skip to content

Codex CLI: add persistent self-healing monitor workflows for long-running jobs #32993

Open

Description

@DeepThinkingZhouLiu

What variant of Codex are you using?

Codex CLI / TUI, especially on headless Linux, SSH, and remote development machines.

What feature would you like to see?

Summary

Add a first-class, persistent monitor workflow to Codex CLI that can return to the same thread at a future time or when a background event occurs, inspect progress, and perform a bounded diagnose -> fix -> restart/resume -> verify loop when it detects a failure.

This is more than a timer or status reminder. The goal is to let Codex safely supervise long-running development, deployment, evaluation, and ML training jobs without requiring the user to repeatedly ask for progress updates.

Problem

A typical workflow today looks like this:

  1. Codex starts a long-running build, experiment, training run, deployment, or development task.
  2. The user leaves the session and returns 30 minutes later.
  3. Only after the user asks for status does Codex inspect the logs and discover that the job crashed, stalled, produced bad cases, or hit a bug much earlier.
  4. Time was lost because the agent had no durable way to wake itself, check progress, and continue remediation.

External cron jobs can inspect a process or log, but they do not naturally preserve the active Codex thread, its context, constraints, prior diagnosis, sandbox policy, and repair history.

Example workflows

Illustrative commands:

/monitor in 30m "Check the training run, summarize progress, and investigate any failures."

/monitor every 30m --until complete --name training-guard \
  "Inspect logs and metrics. If the run is healthy, keep waiting. If it crashed, stalled,
   regressed, or produced bad cases, diagnose the cause, apply fixes within the approved
   scope, restart or resume the run, verify recovery, and continue monitoring."

The exact syntax is not important; the end-to-end behavior is.

Desired lifecycle

start job
  -> sleep without model calls
  -> wake on a timer or relevant event
  -> inspect process, logs, metrics, tests, and artifacts
  -> healthy: record status and wait again
  -> unhealthy: diagnose -> fix -> restart/resume -> verify
  -> completed: stop the monitor and report the final result
  -> unsafe, ambiguous, or permission-blocked: ask the user

Requirements

  1. Time- and event-based wakeups
    • One-shot checks such as "in 30 minutes".
    • Recurring checks such as "every 30 minutes".
    • Optional event triggers such as process exit, matching an error pattern, file changes, CI completion, metric thresholds, or stalled progress.
  2. Same-thread continuity
    • Resume the same Codex thread with its existing context and constraints.
    • Do not start an unrelated stateless codex exec run for every check.
  3. No token-burning polling while idle
    • Waiting should make no model calls.
    • Local timers or event watchers should wake the model only when a check is due or a relevant event occurs.
  4. Bounded autonomous remediation
    • On a detected failure, Codex may inspect evidence, make changes within the current sandbox and explicitly approved scope, run targeted validation, and restart or resume the job.
    • Preserve normal approval boundaries for privileged, destructive, publishing, or otherwise high-impact actions.
    • Support maximum retries, time/cost budgets, and fail-closed stop conditions to prevent runaway self-repair loops.
  5. Durability and delivery safety
    • Ideally persist monitor state across terminal or app-server restarts.
    • Do not lose completion or failure events while the thread is busy or temporarily unavailable.
    • Do not interrupt an active turn; queue or coalesce the wakeup for the next safe point.
  6. Inspectable control surface
    • List active monitors, next wake time, latest check, health state, and last action.
    • Pause, resume, trigger now, edit, and stop a monitor.
    • Keep a bounded history of checks, detected anomalies, repairs, validation results, and termination reason.

Why this should be an end-to-end feature

Several existing issues cover important primitives:

Those primitives are necessary, but this request focuses on the missing user-facing workflow and lifecycle contract: monitor -> detect -> diagnose -> remediate -> verify -> continue or stop, with persistent same-thread context and explicit safety boundaries.

If the underlying implementation is split across the scheduler, monitor, goal, and background-exec components, this issue can serve as the end-to-end CLI behavior and acceptance criteria tying them together.

Additional information

The strongest use cases are long-running training and evaluation jobs, development servers, CI or deployment monitoring, overnight test matrices, and other workflows where discovering a failure only after the user manually asks for progress wastes substantial time.

Activity

  1. github-actions commented on Jul 14, 2026

    @github-actions
    Contributor

    Potential duplicates detected. Please review them and close your issue if it is a duplicate.

    Powered by Codex Action

  2. xyjigsaw commented on Jul 16, 2026

    @xyjigsaw

    cannot agree more, I need this feature:

    Start background task → session becomes idle → zero-token waiting → task exits → original thread wakes up → Codex continues automatically

  3. iltranqui commented on Jul 19, 2026

    @iltranqui

    Like the TaskMonitor in Claude Code

  4. matebenyovszky commented on Aug 23, 2026

    @matebenyovszky

    Four things from building this loop as a library, in case they save someone a round:

    • Budget the session, not the attempt. "3 retries" per intervention is not a bound — the monitor comes back, sees a fresh failure, and spends 3 more. The budget has to belong to the job+failure pair, or you get a polite infinite loop.
    • Verify before touching the live job. "It restarted and hasn't crashed yet" is a delay, not a verification. Run the candidate in isolation first, or the monitor's own intervention becomes the next thing to diagnose.
    • Define the give-up state. Restore whatever was modified, and report the ORIGINAL failure — not the later failure of the monitor's own candidate. Reporting the second was the most confusing bug we shipped: the user sees an error unrelated to their job.
    • Classify before diagnosing. OOM, rate limit, preempted spot instance and a deterministic code bug all look like "the job died". Restart-and-resume fixes the first three and loops forever on the fourth.

    Full propose → verify → apply → restore spec, as prior art: https://github.com/matebenyovszky/healing-agent/blob/main/docs/apply-verify-design.md

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    CLIIssues related to the Codex CLIautomationsenhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions