Skip to content

Make import retry safe after ambiguous queue enqueue outcomes #523

Open

Description

@coderabbitai

Requested by @mahmoud-samy as a follow-up to PR #519.

Problem

In packages/trpc/src/routers/import.ts, importRouter.retry changes an import from failed to pending before calling the BullMQ importQueue.add. If add rejects, the route restores failed, which addresses a definite enqueue failure. If Redis persisted the job but the acknowledgement was lost, the rollback can permit a second retry and concurrent jobs for the same import ID. Concurrent jobs can interfere with staging cleanup by import ID in apps/worker/src/jobs/import.ts. A pending import with no live job may also need recovery.

Proposed work

  • Investigate this repository's BullMQ version and verify duplicate-job behavior for a deterministic jobId keyed on the import ID. Account for job lifecycle and later retries so valid retries remain possible.
  • Make retry enqueue idempotent across ambiguous acknowledgements. Do not reset an import to failed when a job might already have been persisted without confirming the outcome.
  • Assess whether a reconciliation process is necessary for pending imports that lack a live job. Implement one if needed, with a safe way to distinguish live, completed, and missing jobs.
  • Add tests for definite enqueue failures, ambiguous enqueue outcomes, duplicate retries, and reconciliation if implemented.

Acceptance criteria

  • An ambiguous acknowledgement cannot cause two active import jobs for the same import ID.
  • A definite enqueue failure remains retryable.
  • Pending imports without a live job have a documented recovery path.
  • Tests cover the selected recovery and idempotency behavior.

Related PR: #519
Original review comment: #519 (comment)

Activity

  1. avinash-git-24 commented on Sep 28, 2026

    @avinash-git-24

    Hey team, I can make importRouter.retry in packages/trpc/src/routers/import.ts check importQueue.getJob(jobId) before rolling status back to failed on ambiguous BullMQ enqueue errors. Mind if I take this up?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions