[client] Metadata initialization failures can leave partition writes stuck #4928

Search before asking

• I searched in the issues and found nothing similar.

Fluss version

0.9-based downstream Java client (based on the artifact checked after the incident; the exact incident-time binary was not retained).

Please describe the bug 🐞

Related work: #483 and #1458. This report covers two related gaps in the same failure chain: metadata initialization can complete without required table mappings, and subsequent partition metadata refresh cannot restore those mappings.

The code paths described below were inspected in v1.0.0 and main at dc3eb23ea8541a39ec5e3aac13a6e65169cd6e54. The original incident occurred in a downstream deployment; we have not yet run an upstream reproducer.

In our application, writer initialization completed without reporting an error, but subsequent writes repeatedly failed with:

IllegalArgumentException: table path not found for tableId <id> in cluster

The failure occurred while parsing partition metadata, before records entered the write accumulator. Runtime logs confirm the missing mapping. An initialization timeout is a suspected trigger, but the historical logs do not establish it directly.

The relevant code paths are:

• MetadataUpdater.updateMetadata() catches timeout and retriable exceptions without propagating them. If the separate Admin getTableInfo() request succeeds, getTable() can return while the table mapping is still missing.
• checkAndUpdatePartitionMetadata(PhysicalTablePath) sends a partition-only metadata request.
• MetadataUtils then needs the parent table’s ID-to-path mapping to process the response. If neither the response nor the cache contains it, parsing fails.

Relationship to #483 and #1458

The configurable timeout and retry support proposed in #1458 could recover a transient initialization failure if a retry obtains and caches the table metadata. However, retry support alone would leave two gaps:

1. Retries are exhausted and the failure is still swallowed. Initialization can proceed with an incomplete cache unless failure propagation or readiness handling also changes.
2. The cache is already missing the table mapping. Repeating a successful partition-only RPC does not fetch the missing parent-table metadata, so response processing can keep failing even after connectivity has recovered.

Both initialization failure propagation and recovery of missing table mappings are needed to address this failure chain. They are included in the scope of this issue.

Proposed reproduction and validation

These are proposed tests, not completed reproductions:

1. Transient initialization failure: Inject a timeout or retriable exception into the initial table metadata request, then allow a retry to succeed within the retry budget. Verify that initialization returns successfully only after the required table mappings are populated, and subsequent writes succeed.
2. Persistent initialization failure: Make the required table metadata requests fail until the bounded retry budget is exhausted. Verify that getTable() propagates an exception with the underlying cause instead of returning a partially initialized table. A successful response from the separate Admin getTableInfo() request must not mask the failure. Also verify that initialization fails if a response still leaves the required mappings missing.
3. Recovery during subsequent refresh: Start with valid server information but no table mapping, while the table and partition exist on the server. Verify that partition refresh requests the missing parent-table metadata, rebuilds the mappings, and allows subsequent writes to succeed. Cover both single-partition and batch refresh paths, with an existing-mapping control case to check that unnecessary table metadata requests are avoided.

The regression tests should also check that recoverable background refresh failures retain appropriate retry behavior.

Solution

1. Require successful metadata initialization, or throw after bounded retries. The table initialization path, including getTable(), must obtain and validate the metadata required for that table before reporting success. Retry transient failures within a finite attempt or time budget. If retries are exhausted or required mappings remain unavailable, propagate an exception to the caller with the underlying cause. A successful independent getTableInfo() request must not turn an incomplete metadata initialization into a successful result.

2. Restore missing table mappings during subsequent metadata refresh. When refreshing partition metadata, detect missing parent-table mappings and include the corresponding table metadata in the request, or fetch it before processing the partition response. Update the cache from the complete response so that later writes can recover. Cover both single-partition and batch refresh paths.

updateMetadata() is shared by initialization and background refresh paths. The implementation should distinguish their requirements: initialization must wait for required metadata or fail, while background refresh should retain appropriate recovery and retry behavior for transient failures.

I’m willing to contribute a reproducer, regression tests, and fixes for both parts, and coordinate with the related metadata refactoring work tracked by #483 and #1458.

Are you willing to submit a PR?

• I'm willing to submit a PR!