Page-level ConcurrentModificationException on bulk edge CREATE_UPDATE to a shared single-bucket type under concurrent load (remote HTTP, explicit tx) — reinforces #5279

[ivanfrias]

Summary

This reinforces #5279 with a concrete, reproducible-under-load case.

Under concurrent write load, a bulk edge CREATE_UPDATE (~3,900 edges in one explicit transaction, committed over the remote HTTP client) intermittently fails at commit with a page-level ConcurrentModificationException — even though the records being written are logically unrelated. It is a false conflict in the sense of #5279: no real record overlap, just writers colliding on the same page under concurrency.

Run in isolation the same operation always succeeds; it only fails when multiple independent clients write concurrently.

Environment

• ArcadeDB 26.8.1-SNAPSHOT
• 3-node HA cluster (writes serialize on the leader)
• Java remote client (RemoteDatabase), explicit begin / multi-batch / commit
• Edges addressed directly by record id (g.E(<rid>)) — no outE traversal, so this is not one of the super-node edge-append cases already improved in 26.7.2 / 26.8.1-SNAPSHOT.

Error (client side, returned from remote commit)

com.arcadedb.exception.ConcurrentModificationException: Concurrent modification
on page PageId(<db>/343/0) in file '<Type>_14.343.65536.v0.bucket'
(current v.882 <> database v.884). Please retry the operation (threadId=95)
    at ... RemoteDatabase.commit
    at ... RemoteHttpComponent.manageException

We are already using the mitigations usually suggested — and they are not enough

This is the key point for #5279: the usual advice ("retry it" / "add more buckets") is already applied here, and the conflict still isn't recoverable.

• We already use your built-in transaction retry. The whole bulk runs inside database.transaction(txBlock, joinCurrentTransaction=false, attempts=5), and RemoteDatabase.transaction(...) retries the entire begin→commit block on NeedRetryException / ConcurrentModificationException (CME extends NeedRetryException, and executeCommand explicitly flags it // SUPPORT RETRY). The failure above happens after all 5 attempts are exhausted — under sustained concurrency the page keeps being re-versioned between our read and commit, so the retry budget never converges.
• The schema is already heavily bucketed. Every concrete vertex type (5 types) and every concrete edge type (8 types) has 16 buckets. (The base V/E keep the default single bucket, but records live in the 16-bucket concrete subtypes.) So this is not simple type-bucket scarcity.

Where the conflict actually is

The conflicting page lives in a vertex bucket (<Type>_14...), and the workload attaches ~3,900 edges to a small number of hub vertices in a single commit. This looks like super-node edge-list append contention on a hot vertex — adding edges bumps the version of the page holding the vertex / edge-list head — rather than record-level contention. Increasing the type's bucket count does not help a single hot vertex. There is also a UNIQUE index on E(@out, @in) that every edge insert must update.

Questions

1. Given Concurrent edge insertion into a super-node vertex drops edges (edge-list chunk-allocation race) #5147 and the "parallel add edge for super nodes" work mentioned here for 26.8.1-SNAPSHOT: is there a supported/config-gated way (e.g. arcadedb.graph.edgeAppendMerge) to let concurrent edge appends to the same super-node commit without a full-transaction CME? For this workload that seems to be the missing piece — more so than client retries or more buckets.
2. For explicit transactions over the remote HTTP client, is there anything beyond the attempts retry loop we should be doing (the retry param from Concurrent modification error on the same bucket (@v25.3.2) #2236 appears to cover only the implicit /command path)?
3. Is there any plan to raise CME only on true record conflicts rather than page-level collisions (the core ask in Is ArcadeDB ready for a real multi-user environment? The problem of ConcurrentModificationExceptions #5279)?

Related: #5279, #2236, #5147.

[mdre]

That's the concern I foresee for real environment.

[lvca]

This is something we're aware of. The best strategy is to use different buckets on massive inserts with 1 bucket per thread = zero conflict. But more in general we could implement something we were working on a private branch for quite some time: automatic merge page changes in case of conflict on the same page, but different records.

We already have something ready that automatically manages the concurrent insert on the same page. Adding update/delete could be something we could do in the future, but I think having a concurrent insert on the same page already means fixing 80% of the issues.

It doesn't mean you don't need to retry, it will happen at a fraction of the times (in the case the 2nd insert can't find proper space in the page).

[mdre]

Please consider changing the name to ConcurrentPageModificationException and leave ConcurrentModificationException ready to be thrown when a real CME occurs.