[Umbrella][client] Java client robustness and performance improvements #4980

Search before asking

• I searched in the issues and found related work for some items (linked below), but no umbrella issue tracking them.

Description

This umbrella issue tracks robustness and performance improvements to the Java client (fluss-client). They came out of a side-by-side review of the Java client and the Rust client (fluss-rust). In several places the Rust client already handles a failure or edge case that the Java client misses.

Items are grouped by impact.

P0: calls that never return or a stalled connection

• Closing the writer leaves write futures pending. Graceful close stops once the accumulator is drained, so a retriable failure of an in-flight batch is re-queued into a queue nothing reads. Force close aborts nothing unless there is a fatal error. [client] Writer close leaves in-flight and pending writes unfinished #5008, [Bug] Memory was leaked by query when close sender before send WriteBatch. #321 (PR [client] Complete pending writes when the writer is closed #5009)
• Operations on a dropped table never complete. The metadata refresh fails with "table does not exist", which is only handled for missing partitions. Writes stall the sender for every table on the connection and flush() hangs. Lookups are re-queued forever. [client] Lookups on a dropped table never return #4706 (PRs [client] Fail lookups and writes on a dropped or re-created table instead of hanging #4757, [server] Fix stale metadata served for missing and re-created tables #4758); related: [server] Stale or wrong metadata served for missing and re-created tables #4755.

P2: robustness

• No backoff on metadata refreshes for buckets without a known leader. The writer does a blocking metadata round trip on every sender pass during a leader election. Lookups refresh once per pending lookup and sleep a fixed 100 ms. A missing partition also fails the refresh for other partitions. [client] Lookups for a leaderless bucket retry without backoff #4971 (PR [client] Back off lookups for buckets without a leader #4991), [client] A missing partition fails the metadata refresh of other unknown-leader partitions #4975
• A re-created table leaves stale partition metadata in the client cache. Metadata updates merge new entries over old ones and never remove anything, so partition and bucket entries from the dropped table survive. [server] Table and partition metadata may become inconsistent after drop and recreation #4741, clearup useless fluss partition info in writer metadata cache #3362
• Security token renewal can spin. An expired token or clock skew gives a negative renewal delay, which renews immediately in a loop. [client] Prevent invalid security-token renewal scheduling #5012 (PR [client] Prevent tight-loop security-token renewal #5013)
• Arrow read context leak in LimitBatchScanner. [client] LimitBatchScanner leaks Arrow read context on log-table LIMIT scan #4353 (PR [client] Close Arrow read context after LIMIT log scan #4356)
• Small cleanups. Swapped SASL option descriptions, lastPollSecondsAgo before the first poll, and lost unknown error codes. [client] Small cleanups: swapped SASL option docs, lastPollSecondsAgo before first poll, and unknown error codes #5021

P3: features available in the Rust client

• Public Arrow batch scanner that supports projection. [client] Expose Arrow changelog batches for primary-key tables #4005; related: [client] Push projection into Arrow loading for remote log scans #4882.
• Bounded log reads. [client] Support bounded Log Table scanning #4544