Repository navigation
[server] Lookups queue for seconds behind slow putKv requests in the shared request worker pool #4926
Description
Search before asking
- I searched in the issues and found nothing similar.
Motivation
On Fluss 1.0.0, lookups and prefix lookups spend almost all their latency waiting in a request queue, not executing, even though most request workers are idle.
Each client connection is assigned to one worker, and each worker processes its queue one request at a time. A putKv holds the bucket's KV write lock while it reads the old value of every record from RocksDB (read-before-write). When the KV is much larger than the block cache these reads go to disk, so the worker serving a write-heavy connection is busy almost all the time, and every request queued behind it waits, including lookups from other clients.
Note: we see this under sustained heavy write load (a large backfill), not in steady state.
Observed (per tablet server, 5-minute windows, while Flink delta joins and CDC upserts run together, 24 worker threads):
| Request | Queue time p95 / p99 | Process time p95 / p99 |
|---|---|---|
prefixLookup |
2.1 s / 5.9 s | 7 / 12 ms |
lookup |
12.9 s / 20.8 s | 38 / 61 ms |
- The backlog sits on one or two workers: on one server, a single worker had 37–210 requests queued in every sample while 16–23 of the 24 queues were empty.
- Thread dumps show that worker in
putKv, reading old values from RocksDB, in 11 of 13 samples, while 14–23 of the 24 workers were idle. - The KV write lock spreads the blocking: a follower's replication fetch on another worker was blocked on the same bucket's lock (4 of 13 samples), stalling that worker's queue too.
- Raising
netty.server.num-worker-threadsfrom 8 to 24 didn't help: it adds workers, but a write-heavy connection is still tied to one of them. - Tablet CPU (0.1–0.5 of 4 cores) and disk (5–7k of 10k IOPS) are far from saturated.
- Lookup latency is set by the slowest write traffic sharing the worker. Lookup-heavy workloads (delta joins, lookup joins) stall while large writes run.
Related but different: #4057, #4767/#4768 and #4749 reduce KV lock contention, i.e. lookup processing time, which is already low here. This issue is about lookups waiting behind writes in the queue.
Solution
- Don't queue lookups behind the connection's writes: dispatch
lookupandprefixLookupto a separate read pool, or to any free worker. Lookups have no ordering requirement. Writes (putKv,produceLog) stay on their connection's worker, which idempotent writes with several in-flight batches per bucket rely on. - Shorten how long
putKvholds the KV write lock by batching the old-value reads into one MultiGet ([kv] Batch old-value lookups in the KV write path #4063/[kv] Batch old value lookups in write path #4065). This is complementary: it shortens the blocking, but without (1) any long-running write can still block reads queued behind it.
Replication traffic shares these queues with client traffic; isolating the internal listener (#4856/#4857) would keep follower fetches out of them. Per-worker busy time and per-request-type queue metrics would also make this easier to diagnose.
No response
Anything else?
No response
Willingness to contribute
- I'm willing to submit a PR!
Activity
polyzos commentedon Oct 28, 2026
@wuchong @platinumhamburg before implementing anything, can you let me know your thoughts on this?
[-]server] Lookups queue for seconds behind slow putKv requests in the shared request worker pool[/-][+][server] Lookups queue for seconds behind slow putKv requests in the shared request worker pool[/+]platinumhamburg commentedon Oct 29, 2026
Hi @polyzos , Thanks for sharing the observations. I agree that connection-bound request queues can cause head-of-line blocking, and separating reads from writes is worth evaluating. A few thoughts:
The blocking model is fairly complex. Besides the long synchronous execution of
putKvandprefixLookup, lock contention is another major source of worker blocking. Separate pools can reduce queue interference, but they do not remove lock waits; blocked reads could still occupy the read pool. Longer term, I think we should look at fair request scheduling together with bucket affinity, while preserving the required write ordering.Could you share whether RocksDB shared memory is enabled, along with the block-cache and index/filter settings? Given the reported lookup latency, I would check cache pressure as well. If shared memory is enabled, I would strongly recommend enabling partitioned indexes/filters and ensuring the shared block cache is adequately sized. Keeping index/filter blocks outside the shared block cache is another option, provided their memory is budgeted separately. This is a diagnostic hypothesis; the current observations do not establish cache configuration as the cause.
I have also been working on related optimizations that may complement this:
These PRs are still open. The related prefix-lookup lock reduction in [kv] Iterate prefix lookups outside the KV lock #4749 has already merged..
For read/write pool separation, could we run a controlled production trial to quantify the benefit? Comparing queue time, processing time, lookup tail latency, and write throughput under comparable load would help, ideally keeping the total worker budget unchanged and distinguishing same-bucket contention from unrelated-bucket traffic. Also,
lookupwithinsertIfNotExistshas write semantics and should be classified as a write request.polyzos commentedon Oct 29, 2026
Thanks @platinumhamburg, this is helpful. 👍
RocksDB config (1.0.0), per tablet server: shared block cache enabled, 3 GB; shared write buffer 1 GB;
cache-index-and-filter-blocks: truewith high priority and the block size is 4 KB.The shared block cache is at 100% (3.21 / 3.22 GB, 0 pinned), with roughly 230 GB of KV per server. RocksDB get p99 is 6–20 ms on the large tables, vs ~1–2 ms on recently written ones. So a cache miss likely costs several reads (index/filter, then data).
That would explain why each
putKvholds the worker and the lock for so long, so it makes the queueing worse without being its cause.I'm running various experiments to better to fully understand different cases and scenarios - we may as well just scale the number of buckets and number of TS. - I just want to track this because long-term it might end up being a pretty useful improvement.
Fair scheduling with bucket affinity sounds like a great long-term direction 👍
platinumhamburg commentedon Oct 29, 2026
Hi @polyzos, thanks for sharing the details. With roughly 230 GB of KV data per TS and a 3 GB shared block cache, I strongly suspect index/filter cache churn contributes significantly to the latency. Metadata misses introduce additional block reads, I/O, and decompression, extending worker occupancy and lock hold times.
I strongly recommend trying partitioned indexes/filters from #4612:
This takes effect when opening a RocksDB instance. Existing SSTs gradually adopt partitioned metadata as compaction rewrites them, which should improve cache efficiency if metadata churn is the bottleneck.
#4612 may not be included in your 1.0 build, so please check your version. If convenient, you could test main, or cherry-pick the recent primary-key table optimizations onto 1.0 and build a test image.
I also strongly recommend testing #4857 with separate internal and external listeners. It isolates replication and control traffic from client queues and backpressure. In our heavy mixed write and Delta Join workload, it substantially reduced ISR instability and brought
putKvprocessing p99 down from seconds to milliseconds. It would be useful to see whether your workload benefits similarly.