publishPages blocks the committers of every database inside the JVM-wide page-manager lock whenever the flush queue fills

[lvca]

Found while fixing #6200, which removed one trigger for this and left the other in place. #6200 was "one suspended database's backlog stops the flush thread for every other database"; this is the same stall reached without any suspension at all.

The shape

PageManager.publishPages takes the JVM-wide page-manager lock and calls scheduleFlushOfPages inside it (engine/src/main/java/com/arcadedb/engine/PageManager.java, lock() → writePagesNoBackpressure → scheduleFlushOfPages). That method blocks:

// TRY TO INSERT THE PAGE IN THE QUEUE UNTIL THE THREAD IS STILL RUNNING
while (running) {
  if (queue.offer(new PagesToFlush(pages), 1, TimeUnit.SECONDS))
    return;
}

So whenever the bounded flush queue (arcadedb.pageFlushQueue, default 512, 8 under the low-ram profile) is full, the committing thread parks in offer holding a lock every other committer in the process needs. The queue fills whenever commits outrun the disk for a moment: a write burst, a slow or contended volume, an fsync spike, a compaction landing at the wrong time. No suspension, no backup, no HA involved.

The result is that one database's write burst serializes the commits of every unrelated database in the JVM - including databases on a different, idle volume, whose pages could have been written immediately.

Why this is worth separating from the queue's job

The bounded queue is meant to backpressure the writers it belongs to; that part is right. What is wrong is only that the wait is served while holding a JVM-wide lock, so it is charged to everybody. This is exactly the distinction #6200 drew for the deferred-RAM cap - "the ceiling is global because heap is; the response must not be" - and the fix there was to move the wait ahead of lock(). That precedent applies here without change of principle.

Why the lock cannot simply be released around the enqueue

publishPages holds the lock across BOTH halves of publication - the synchronous page write and the flush enqueue - and the snapshot t0 barrier (#6075/#6125) depends on exactly that: step 3 of openSnapshot takes the same lock precisely so that "from here on no committer can put a page on disk OR into the flush pipeline", which is what makes its residual drain converge by construction. Dropping the lock between the two halves reintroduces the gap that #6125 removed, and with it the retry loop and the inexact lastTxId it was written to fix. So the enqueue has to stay inside the lock, which means the WAIT has to move outside it.

Proposed

The shape #6200 already established: admission control before the lock.

1. Before lock(), wait (interruptibly, bounded by running) until the flush queue has room - "room" meaning at least one free slot, so the enqueue inside the lock is then effectively non-blocking.
2. Keep the blocking offer inside as the safety net it is today, since a racing publisher can still take the last slot; it just stops being the normal way the queue is entered.
3. Signal the waiters from the flush thread when it polls a batch, rather than having them poll - ArrayBlockingQueue already gives this for free if the wait is expressed as a permit/semaphore sized to the queue, which also removes the "wake up, find the slot taken, sleep again" thundering herd.

An alternative worth measuring first: make the queue per-database. It bounds each database's backlog independently and removes the cross-database coupling at the source, but it does not on its own fix this bug - a committer that blocks on its OWN full queue while holding the JVM-wide lock still stalls everyone - so it would have to be combined with step 1 regardless.

Verification

• Two databases, A committing a burst large enough to fill arcadedb.pageFlushQueue against a deliberately slow disk (or with the flush thread paused), B committing one small transaction: today B's commit waits behind A's; it should not. The white-box shape used by Issue6200PerDatabaseDeferredBacklogTest fits directly.
• A test that pins the invariant rather than the symptom: while a committer is waiting for queue room, another thread must be able to take the page-manager lock (PageManager.executeInLock).
• Issue6125SnapshotBarrierAndCapTest must stay green - it is what proves the publication lock still covers the enqueue.