apache / apache/fluss

[Server] Clean leader reassignment can leave divergent WAL in an ISR replica

Open
#4,269 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.

### Fluss version

0.9.0 (latest release)

### Please describe the bug 🐞

A clean leader reassignment can leave replicas with different records at the same offsets, while the new leader acknowledges writes with `acks=-1` and all replicas remain in the ISR.

This was reproduced using a regular Log Table without secondary indexes.

**Steps to reproduce**

Use three replicas, A, B, and C, with A as leader and ISR `[A, B, C]`. All offset ranges below are half-open.

1. Append and commit a common prefix `[0,10)` on all three replicas.
2. Append another batch `[10,15)` to A with `acks=1`. Deliver its replication response only to C. B remains at offset 10 and stays in the ISR; A's HW remains 10.
3. Perform a clean leader reassignment to B using `ReassignmentLeaderElection`.
4. Append different records to B. Since B's LEO is 10, these records occupy `[10,15)`.
5. Resume replication from B to A and C.

**Actual behavior**

The write in step 4 completes successfully with `acks=-1`, but the replicas contain:

| Replica | HW | Records at offsets 10–14 |
|---|---:|---|
| A | 15 | New records from B |
| B, current leader | 15 | New records |
| C, still in ISR | 15 | Old records from A |

The previously committed prefix `[0,10)` remains intact. The inconsistency affects the replacement records acknowledged by the new leader.

### Solution

The replication protocol needs to establish a common log prefix when leadership changes, rather than treating matching offsets as proof of matching history. Leader-epoch-based reconciliation and the relationship between follower HW and local LEO need to be considered together.

Append offset-continuity checks alone would not detect this case because the divergent records occupy the same offsets.

### Related work
Tracked under #673.

This issue documents a reproducible WAL divergence during a normal,
clean ISR leader migration: a write can be acknowledged while ISR
replicas retain different records at the same offsets.

Related implementation work: #778, #788, and #789.

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with ReassignmentLeaderElection and the replication protocol, then reproduce the three-replica sequence described in the issue. Trace the leader change, follower HW, and local LEO through the reassignment; done means a clean migration establishes a common log prefix before an acks=-1 write is acknowledged.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.