Skip to content

Queue dedupe is a one-way door: a cancelled task permanently blocks re-enqueue of its issue #1298

Closed

Description

@dezren39

Bug (live instance found 2026-10-26, monitor run 23:19 CDT)

agent.enqueue_repo_task (migration 0008) inserts with
on conflict (dedupe_key) do nothing and then returns the existing
row — for any status, including cancelled. Combined with the
unique index on dedupe_key, a cancelled task permanently owns its
issue's work identity: the issue can never be enqueued again, and the
caller cannot tell, because enqueue returns the cancelled row as if
it had succeeded.

Live instance

  • Issue id(ci): F11.4 fast CI lane under 15 minutes #551 (id-check flake leg exceeds its 60-minute cap) had an
    inert shadow task 1064 (attempts 0, never worked).
  • The fleet monitor's safety net built a live triage packet for id(ci): F11.4 fast CI lane under 15 minutes #551
    and was blocked by the dedupe wall; the operator cancelled 1064
    (verified cancelled, reason superseded) and retried the enqueue.
  • The retry returned task 1064 — the cancelled shadow row — with no
    new task created. Verified in agent.tasks: exactly one row exists
    for repo-health:v1:triage:developing-today/code:issue:551, status
    cancelled. id(ci): F11.4 fast CI lane under 15 minutes #551 sits ownerless, and every future monitor run will
    repeat the same silent no-op.

Why it matters

The monitor's Duty 1 promise is "a filed issue with the work-order
label must never sit ownerless." This bug breaks that promise
silently for any issue whose task was ever cancelled — and the
fleet cancels tasks routinely (orphans, superseded packets,
issue-closed cleanups; e.g. nix task 1172 on 2026-10-26). Any of
those issues reopening hits the same wall.

Proposed fix (pick one, migration + synthetic test like 0012)

  • (a) Revive on enqueue (recommended): on conflict where the
    existing row is cancelled, update it back to queued with the
    new payload/priority/not_before, reset result/last_error, and write
    an enqueued event with revived_from_cancelled: true. Smallest
    schema change; keeps one row per identity; audit trail preserved.
  • (b) Partial unique index on dedupe_key covering live statuses
    only — allows a second row per identity; weakens the "dedupe key
    names the task" lookups the monitor relies on.
  • (c) An explicit reopen_repo_task verb — works, but every producer
    must learn to call it after a failed enqueue, and the silent-no-op
    shape remains for anyone who doesn't.

Decide whether failed tasks should revive the same way (probably
not — failure is a worked outcome; cancellation is an operator
decision. Record the reasoning either way).

Acceptance criteria

  • Synthetic test: enqueue → cancel → enqueue same key returns a
    live queued task (revival event present); enqueue →
    enqueue (live) still returns the existing row unchanged.
  • id(ci): F11.4 fast CI lane under 15 minutes #551's triage packet enqueues successfully to
    repo-health-code-pilot and is claimable.
  • The helper/monitor can distinguish "enqueued new", "already
    live", and "revived" from the returned row + events.

Child of epic #1233. Filed by Muse (operator) 2026-10-27, from the
fleet monitor's #551 flag.

Activity

  1. added
    kind/bugSomething isn't working (prefixed form of the classic bug label)
    priority/highImportant and time-sensitive
    queue/queuedHas a queued task in the Supabase agent queue
    on Oct 27, 2026
  2. developing-today commented on Oct 27, 2026

    @developing-today
    Contributor
  3. added
    area/infraInfrastructure, deployment, and operations
    effort/sSmall: expected under 2 hours of work
    meta/enrichedIntake review done: acceptance criteria, links, and normalized labels added
    on Oct 27, 2026
  4. dezren39 commented on Oct 27, 2026

    @dezren39
    MemberAuthor

    Correction to the live instance (2026-10-27, Lane B shift): #551 is not wholly ownerless. A small_fix task for the same issue (task 1283) exists under a different dedupe key (the key embeds the packet kind: ...:triage:... vs the small_fix key) and is progressing — parked with measurements posted (comment 6434473452). What remains blocked is specifically the triage/ci-investigation packet, whose key is owned by the cancelled shadow task 1064. The bug and the silent-no-op behavior stand exactly as filed; only the "sits ownerless" framing should read "this packet can never be enqueued."

  5. dezren39 commented on Oct 27, 2026

    @dezren39
    MemberAuthor

    🤖 Agent note · filed

    • Agent: fleet monitor (repo-fleet-monitor)
    • Session / ID: cron repo-fleet-monitor, run 2026-10-27 05:29 CDT
    • Found via: monitor reconciliation
    • Task: Fresh reproduction of this bug, 2026-10-27T10:49Z: issue mcpx: Phase 1 of #1384 — supervised stdio transport behind a flag (supervisor owns the child's pipes) #1385 carries queue/queued with no live pilot task (cancelled shadow task 1354 holds its dedupe key). The monitor's repo-queue.py enqueue --queue repo-health-code-pilot returned task 1354 (queue repo-health-code-shadow, status cancelled) and created nothing in the pilot queue — verified by direct live-queue query immediately after. The issue remains ownerless through no producer's fault. This is the second live consequence class for the one-way door (a wrong-queue first attempt, not just a cancelled pilot task).
  6. added
    work/assignedQueue worker lifecycle state
    and removed
    queue/queuedHas a queued task in the Supabase agent queue
    on Oct 27, 2026
  7. 4 remaining items

  8. added
    queue/queuedHas a queued task in the Supabase agent queue
    and removed
    queue/claimedQueue worker lifecycle state
    work/assignedQueue worker lifecycle state
    on Oct 27, 2026
  9. dezren39 commented on Oct 27, 2026

    @dezren39
    MemberAuthor

    🤖 Agent note · picked up

  10. added
    work/assignedQueue worker lifecycle state
    queue/claimedQueue worker lifecycle state
    and removed
    queue/queuedHas a queued task in the Supabase agent queue
    on Oct 27, 2026
  11. dezren39 commented on Oct 27, 2026

    @dezren39
    MemberAuthor

    🤖 Agent note · done — revive-on-enqueue implemented, applied, and proven (queue task 1392)

    • Agent: queue worker (repo-worker-code), Lane B shift
    • Session / ID: queue task 1392
    • Found via: queue claim
    • Task: Migration 0014_revive_on_enqueue.sql (in ~/workspace/supabase/muse-dev/migrations/, committed ada0e81) re-creates both agent.enqueue_repo_task and agent.enqueue_repo_task_fleet with option (a) from this issue: on conflict where the existing row is cancelled, the row revives in place to queued with the new queue/title/body/payload/priority/not_before/max_attempts, attempts reset to 0, result/last_error/worker/lease/claim/completion state cleared, and an enqueued event is written with revived_from_cancelled: true. Enqueued-new = event without the flag; already-live = no new event and the row returned unchanged; revived = event with the flag — the three cases are distinguishable from the returned row + events.
    • Failed-task decision: failed tasks deliberately do NOT revive — failure is a worked outcome whose accounting (attempts, last_error, result) producers and the monitor rely on; cancellation is an operator decision, so revival matches operator intent. Reasoning is recorded in the migration header.
    • Synthetic test: 0014_revive_on_enqueue_test.sql run live against muse-dev — enqueue → duplicate enqueue returned the same row (task 1466) unchanged → cancel → enqueue returned the same row live queued with the new packet (title/priority replaced, state reset) and the event trail enqueued → cancelled → enqueued (revived_from_cancelled: true). The synthetic task was cancelled as cleanup afterwards. repo-queue.py health read still succeeds.
    • id(ci): F11.4 fast CI lane under 15 minutes #551 acceptance — superseded, not revived: issue id(ci): F11.4 fast CI lane under 15 minutes #551 was CLOSED as completed today (2026-10-27T09:07Z, PR id(ci): split crash-gate into 4 parallel PR checks, test shards 3→4 (#551) #1310 via queue task 1283), so enqueuing its triage packet now would create claimable work for a closed issue. The revival mechanism is proven synthetically above; if id(ci): F11.4 fast CI lane under 15 minutes #551 ever reopens, the next producer enqueue revives row 1064 in place (including into repo-health-code-pilot, since revival adopts the caller's queue).
    • Links: migration + test committed in the muse-dev workspace (ada0e81)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area/infraInfrastructure, deployment, and operationseffort/sSmall: expected under 2 hours of workkind/bugSomething isn't working (prefixed form of the classic bug label)meta/enrichedIntake review done: acceptance criteria, links, and normalized labels addedpriority/highImportant and time-sensitivestatus:done

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions