[server] Apply the planned leader during replica reassignment #3869

Search before asking

• I searched in the issues and found nothing similar.

Description

A replica reassignment plan contains both the target replica order and the target leader. However, the current reassignment flow may finish after applying only the target replica set while keeping the old leader.

For example:

origin: replicas=[0, 1, 2], leader=0
target: replicas=[1, 0, 3], leader=1

The migration may currently progress through:

[0, 1, 2] -> [0, 1, 2, 3] -> [0, 1, 3]

but leave server 0 as leader. The final cluster state then differs from the generated rebalance plan and may preserve the leader imbalance that rebalance intended to remove.

Proposed solution

Apply the planned leader as part of replica reassignment Phase B:

• After all target replicas have caught up, elect the plan's target leader if it is eligible and online.
• Keep the target leader inside the target assignment and ISR throughout the transition.
• Persist the resulting LeaderAndIsr state using the normal coordinator fencing rules.
• Wait for the target leader to acknowledge the final NotifyLeaderAndIsr request before completing the bucket task.
• Make restart and timeout reconciliation idempotent when the replica assignment is already final but the planned leader has not yet been applied or acknowledged.
• Ignore stale responses from an earlier leader, epoch, or rebalance attempt.

Acceptance criteria

• Reassigning [0, 1, 2] with leader 0 to [1, 0, 3] with leader 1 finishes with replicas [1, 0, 3] and leader 1.
• A leader-only plan and a combined replica-plus-leader plan both wait for acknowledgement from the planned leader.
• Coordinator restart at each Phase B boundary resumes without reverting or skipping the leader transition.
• Temporary unavailability of the planned leader keeps the task recoverable rather than recording a false success.
• Tests cover stale old-leader responses and repeated reconciliation.

Non-goals

This task does not introduce general rebalance concurrency or change how the plan is stored.

Willingness to contribute

• I'm willing to submit a PR!