[Server] Avoid falsely completing replica deletion after StopReplica failures #4516

Search before asking

• I searched in the issues and found nothing similar.

Fluss version

0.9.0 (latest release)

Please describe the bug 🐞

Problem

When the CoordinatorServer processes a StopReplicaResponse, replica deletion may be incorrectly considered complete:

1. The response is processed only by iterating over the returned bucket results. If a requested bucket is missing from the response, no failure is recorded for that replica.
2. A failed replica deletion is retried a fixed number of times and then marked as successful, even though the TabletServer never confirmed the deletion.

This may allow table or partition deletion to complete while some replicas have not actually been deleted, causing inconsistent CoordinatorServer and TabletServer states.

Expected behavior

• Match every remote replica deletion request with its corresponding per-bucket response.
• Treat explicit errors, missing bucket results, and duplicate bucket results as failures.
• Move a replica to ReplicaDeletionSuccessful only after receiving an explicit successful response.
• Keep failed replicas in ReplicaDeletionStarted so they continue to block table or partition deletion.
• Never convert a failed replica deletion into a successful deletion after a fixed number of attempts.

Scope

This issue focuses only on the correctness of per-bucket StopReplicaResponse handling.

Transport-level retries and recovery after a TabletServer restart are handled separately by the CoordinatorServer request retry framework and the ReplicaDeletionIneligible recovery flow.

Solution

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!