db-mongodb: concurrent commands at the start of a transaction fail with NoSuchTransaction (251) #18679

Describe the Bug

This issue was written with help from AI (Claude). I've reviewed and reproduced the findings.

On MongoDB, a create sometimes fails with:

MongoServerError: Given transaction number 762 on session a71c02ef-… using txnRetryCounter 0 does not match any
in-progress transactions. The active transaction number is 761
  code: 251, codeName: 'NoSuchTransaction', errorLabels: ['TransientTransactionError']
  at FindCursor._initialize (mongodb/lib/cursor/find_cursor.js)

There are already open issues and PRs for individual symptoms of this (see Related). I think they share one root cause, and I'm opening this so it can be fixed in one place rather than at each call site.

Root cause

Payload can send several commands at the same time on a transaction that hasn't sent its first command to the server yet.

• beginTransaction calls session.startTransaction(). In the Node driver this only changes client state; nothing is sent to the server.
• The server learns about the transaction from the first command, which the driver tags with startTransaction: true and the new txnNumber. The driver tags only the first command it builds. Every later command carries the txnNumber without the tag.
• When several commands are sent at the same time on the session, they can use different pooled connections, so they can arrive out of order. If a command without the tag arrives first, the server has never seen that txnNumber and rejects it with NoSuchTransaction. That's what the error says: 762 was given while the server still has 761 as the latest.

Once the first command has completed, running commands concurrently is safe, so this only happens at the start of a transaction. In practice that means create. update and delete read the document on the session first, so the transaction has already started before anything else runs. That also explains why the workarounds in the related issues help: save as draft and then publish, or run one read in beforeOperation.

It depends on timing and latency. In production on Atlas we see it occasionally; the repro below triggers it reliably against Atlas. We haven't tried a local replica set, where the lower latency may hide it.

Where the concurrency comes from

Any code that sends more than one command at the same time on a new transaction triggers it. I know of these:

1. Paginated find in db-mongodb. Model.paginate() (mongoose-paginate-v2) starts the count and the docs query together and waits on both with Promise.all. If that find is the first command in the transaction, they race. This is our production case: a collection beforeChange hook runs req.payload.find({ where: …, limit: 1, req }), and the stack trace is the docs find losing to the count.
2. Field validation with filterOptions. beforeChange field traversal validates sibling fields and array rows with Promise.all, and each relationship/upload field with filterOptions runs its own find in validateFilterOptions. Here the error is caught and logged (Error validating filter options for collection …), so the user gets a false "invalid relationship" validation error instead of a 500. plugin-multi-tenant adds filterOptions to every relationship that points at a tenant-enabled collection, so multi-tenant setups hit this without writing any filterOptions themselves ([plugin-multi-tenant] Race condition in auto-injected relationship field validation produces false "invalid relationship" errors #16462).
3. User code. Any hook that uses Promise.all over Local API calls sharing req.

The open PRs each fix one source: #17053 makes pagination run one query after the other inside transactions, and #16463 queues the plugin's dataloader finds. Each new source would need its own fix, and source 3 can't be fixed that way at all.

Possible directions

I'd like the maintainers' view on where this belongs. Some options we considered:

• Wait for the first command in the adapter. While a session's transaction hasn't sent its first command, getSession lets one caller through and makes the others wait until that command completes. After that, everything runs concurrently as today. This covers every source, including user code, and doesn't change behaviour outside that short window.
• Start the transaction on the server in beginTransaction. Send a cheap command right after startTransaction(), so the server knows about the transaction before any Payload code runs. That's simpler, but every transaction pays for an extra round trip.
• Retry on TransientTransactionError. The label means the whole transaction is safe to retry, and it is MongoDB's documented way to handle this (session.withTransaction() does it). Payload doesn't retry today (MongoDB TransientTransactionError / WriteConflict is not retried for Payload Local API writes #16766). This fits alongside either option above and isn't a replacement for them: retrying hides the race rather than removing it, and source 2 never throws.
• Fix each call site (fix(db-mongodb): serialize paginated count and docs inside transactions #17053, fix(plugin-multi-tenant): serialize transaction dataloader finds #16463). This works for the known sources, but the problem returns whenever new concurrent code is added.

As a workaround we pass pagination: false to the find in our hook. That makes the first command a single query, and the field validation that follows is safe because the transaction has already started.

Related

• [plugin-multi-tenant] Race condition in auto-injected relationship field validation produces false "invalid relationship" errors #16462: plugin-multi-tenant: race in auto-injected relationship validation gives false "invalid relationship"
  errors (source 2)
• fix(plugin-multi-tenant): serialize transaction dataloader finds #16463: fix(plugin-multi-tenant): serialize transaction dataloader finds (open, source 2)
• fix(db-mongodb): serialize paginated count and docs inside transactions #17053: fix(db-mongodb): serialize paginated count and docs inside transactions (open, source 1)
• MongoDB TransientTransactionError / WriteConflict is not retried for Payload Local API writes #16766: TransientTransactionError / WriteConflict is not retried for Local API writes (open)
• fix: prevent read operations from rolling back the caller's transaction #17748: prevent read operations from rolling back the caller's transaction (merged; another way hooks that read with req affect the transaction)
• fix(db-mongodb): fix pagination with collation in transactions #15990: fix pagination with collation in transactions (merged; earlier session bug in the same pagination code)

Link to the code that reproduces this issue

https://github.com/andershermansen/payload-mongo-txn-race

Reproduction Steps

The repro is the blank template with three collections (src/collections/Repro.ts) and an integration test
(tests/int/transaction-race.int.spec.ts) that runs 100 payload.create calls per collection, one at a time:

• orders: a beforeChange hook runs a paginated, filtered find with req (source 1).
• orders-unpaginated: the same hook with pagination: false, as a control.
• posts: an array of relationships with static filterOptions (source 2).

1. Set DATABASE_URL in .env to a MongoDB Atlas cluster.
2. pnpm install
3. pnpm test:int

Expected: every create succeeds. Actual, against Atlas:

• orders fails with NoSuchTransaction (251), as in production.
• posts fails with false validation errors such as
  The following fields are invalid: Rows 2 > Tag, Rows 4 > Tag, and each one logs
  Error validating filter options for collection tags with the 251 error.
• Row 1 never fails. Over all posts failures, the invalid rows were 2 (87 times), 3 (91), 4 (81) and 5 (74), but never 1. Row 1's query is built first, so it is the one that carries startTransaction: true.
• Running one create at a time fails at least as often, so the race happens inside each create and doesn't need other creates running at the same time.

Which area(s) are affected?

db: mongodb

Environment Info

payload: 3.90.2 (repro), 3.90.1 (production)
@payloadcms/db-mongodb: 3.90.2
@payloadcms/plugin-multi-tenant: 3.90.1 (production only)
mongoose: 8.24.1
mongoose-paginate-v2: 1.9.4
mongodb (driver): 6.20.0
Node: v24.21.0
MongoDB: Atlas free tier (M0), 8.0.34