[rebalance] Support preferred leader election goal #4086

Search before asking

• I searched in the issues and found nothing similar.

Motivation

When a TabletServer becomes unavailable, Fluss automatically elects another in-sync replica as the leader.

For example, the initial leader distribution may be:

TS0: 4
TS1: 4
TS2: 4

After TS0 becomes unavailable, its leaders are transferred to the remaining in-sync replicas:

TS0: 0
TS1: 6
TS2: 6

When TS0 recovers and its replicas rejoin the ISR, the affected buckets are already in the Online state, so Fluss does not automatically elect their original first replicas again.
The leader distribution may therefore remain at 0/6/6.

The existing LEADER_DISTRIBUTION goal is not equivalent to preferred leader election:

• It balances the total number of leaders across TabletServers.
• It does not guarantee that the first replica in each bucket assignment becomes the leader.
• It may select a different globally balanced leader placement.
• It may fall back to replica movement when leadership movement alone cannot satisfy the distribution goal.

Operators currently have no deterministic and leader-only operation to restore leadership to the first replica in each bucket assignment.

Solution

Introduce a new rebalance goal:

PREFERRED_LEADER_ELECTION

It can be triggered through the existing Admin API:

admin.rebalance(
Collections.singletonList(GoalType.PREFERRED_LEADER_ELECTION));

and through Flink SQL:

CALL sys.rebalance('PREFERRED_LEADER_ELECTION');

For each bucket:

1. Define the preferred leader as the first replica in the persisted replica assignment, assignment[0].

2. If the current leader is already the preferred leader, do nothing.

3. Otherwise, generate a leader-only rebalance plan when the preferred replica:

   • is alive;
   • is online;
   • is in the current ISR;
   • is not tagged TEMPORARY_OFFLINE or PERMANENT_OFFLINE.

4. Elect exactly the preferred replica as the target leader.

5. Do not fall back to another replica if the preferred replica becomes ineligible.

6. Do not change the replica set or perform replica movement.

Buckets whose preferred replica is not currently eligible should remain unchanged and must not be reported as successfully migrated to the preferred leader.

Anything else?

Goal interaction

For the first phase, PREFERRED_LEADER_ELECTION should only be accepted as a standalone goal.

For example, the following request should be rejected:

CALL sys.rebalance(
'LEADER_DISTRIBUTION,PREFERRED_LEADER_ELECTION'
);

LEADER_DISTRIBUTION and PREFERRED_LEADER_ELECTION can have conflicting objectives: one optimizes global leader counts, while the other deterministically restores each bucket to
assignment[0].

Scope

The first phase is intentionally cluster-wide and manually triggered.

The following are not included in this issue:

• automatic preferred leader election after TabletServer recovery;
• periodic detection or automatic failback;
• table-, partition-, or bucket-level filtering;
• replica reassignment;
• under-replicated bucket repair;
• leader load or traffic-based optimization.

These capabilities can be introduced separately later.

Acceptance criteria

• A bucket whose current leader differs from assignment[0] is moved to assignment[0] when that replica is alive, online, and in the ISR.
• A bucket is left unchanged when its preferred replica is unavailable or outside the ISR.
• The election never falls back to a non-preferred replica.
• The replica assignment and replica count remain unchanged.
• The operation uses only leader-only rebalance tasks.
• Completion waits for a successful acknowledgement from the target preferred leader.
• The operation is idempotent: running it again after completion generates no additional leader movement.
• Requests that combine this goal with another rebalance goal are rejected in the first phase.
• Unit and integration tests cover recovery, unavailable preferred replicas, offline tags, idempotency, and assignment preservation.

Related work

• Support generate rebalance plan #1396 mentioned PreferredLeaderElectionGoal during the early rebalance POC, but it was not included in the final public goals.
• [server] Support generate and execute reblance plan #1452 introduced the rebalance plan generation and execution framework.
• [server] Make leader-only rebalance tasks execute sequentially #3071 made leader-only rebalance tasks execute sequentially.
• [server] Require new leader ack for leader-only rebalance #3556 requires acknowledgement from the target leader before completing a leader-only task.
• [server] Apply the planned leader during replica reassignment #3869 ensures that replica reassignment applies the leader selected by the generated plan. It does not generate preferred leader plans based on assignment[0].

Willingness to contribute

• I'm willing to submit a PR!