[server] Slow snapshot directory creation blocks KV writes #4953

Search before asking

• I searched existing issues and found no report of this specific lock-scope problem.

Fluss version

main (development)

Please describe the bug 🐞

Periodic snapshots call KvTabletSnapshotTarget.initSnapshot() while holding the tablet's KV write lock. This includes a remote mkdirs() call to prepare the exclusive snapshot directory.

Slow remote directory creation therefore blocks puts, lookups, and flushes on the same tablet. Other buckets have separate KV locks, but their snapshot triggers can also be delayed by the shared snapshot scheduler.

A deterministic local reproduction uses a real KvTablet:

1. Write and flush k1.
2. Trigger a periodic snapshot and block remote mkdirs() with a latch.
3. Submit a concurrent write of k2.
4. Observe that the write cannot complete until mkdirs() is released.

Expected: put, lookup, and flush should complete while remote directory creation is blocked. After releasing the latch, the snapshot must still restore the captured state containing only k1, with the corresponding offset, row count, and auto-increment ranges.

Solution

Keep TabletState and the native RocksDB checkpoint captured under the same lock, then initialize the remote directory in the asynchronous snapshot phase. Handle directory initialization failures through the existing asynchronous abort and cleanup path.

Related: #4489 and #4492 address rescheduling after initialization failures; they do not remove remote directory I/O from the tablet lock.

Are you willing to submit a PR?

• I'm willing to submit a PR!