[rust] Per-row appends repeat lookups and contend on map locks #4477

Search before asking

• I searched in the issues and found nothing similar.

Motivation

The Rust row-append path repeats work even after the table, bucket assigner and write queue are initialized.

RecordAccumulator::append() uses write_batches.entry(...).or_insert_with(...) and a mutable entry to retrieve the bucket queue. This takes an exclusive DashMap shard lock on every append, including when nothing needs to be inserted. Producers writing different buckets of the same physical table therefore contend before reaching their separate bucket-local queue mutexes.

There is also repeated per-row work around that lookup:

• WriterClient looks up the bucket assigner by physical path, then the accumulator looks up the queue context using the same path.
• The accumulator repeats a table lookup already performed by WriterClient against the same cluster snapshot.
• Existing-batch appends fetch dynamic batch configuration and clone the compression estimator even though they do not need to create a batch.
• An unpartitioned AppendWriter allocates the same immutable physical path for each row.

The unnecessary exclusive access can be reproduced without a throughput benchmark: initialize a bucket queue, hold a shared read guard on its table entry, and append to the existing bucket from another thread. The append waits for that read guard because it requests exclusive map access.

These are client-side costs on ordinary row writes, including the no-callback path. The required serialization of writes to the same bucket is a separate concern.

Solution

Separate existing-queue lookup from initialization, reuse immutable or already-resolved state, and avoid separate routing and queue-context lookups. Preserve bucket-local locking, queue identity, routing policy and buffer backpressure. The accompanying PR will describe the implementation and validation.

Willingness to contribute

• I'm willing to submit a PR!