[Server] Clean leader reassignment can leave divergent WAL in an ISR replica
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 625
- Avg merge
- 3d 14h
- Merged PRs (30d)
- 97
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
### Fluss version
0.9.0 (latest release)
### Please describe the bug 🐞
A clean leader reassignment can leave replicas with different records at the same offsets, while the new leader acknowledges writes with `acks=-1` and all replicas remain in the ISR.
This was reproduced using a regular Log Table without secondary indexes.
**Steps to reproduce**
Use three replicas, A, B, and C, with A as leader and ISR `[A, B, C]`. All offset ranges below are half-open.
1. Append and commit a common prefix `[0,10)` on all three replicas.
2. Append another batch `[10,15)` to A with `acks=1`. Deliver its replication response only to C. B remains at offset 10 and stays in the ISR; A's HW remains 10.
3. Perform a clean leader reassignment to B using `ReassignmentLeaderElection`.
4. Append different records to B. Since B's LEO is 10, these records occupy `[10,15)`.
5. Resume replication from B to A and C.
**Actual behavior**
The write in step 4 completes successfully with `acks=-1`, but the replicas contain:
| Replica | HW | Records at offsets 10–14 |
|---|---:|---|
| A | 15 | New records from B |
| B, current leader | 15 | New records |
| C, still in ISR | 15 | Old records from A |
The previously committed prefix `[0,10)` remains intact. The inconsistency affects the replacement records acknowledged by the new leader.
### Solution
The replication protocol needs to establish a common log prefix when leadership changes, rather than treating matching offsets as proof of matching history. Leader-epoch-based reconciliation and the relationship between follower HW and local LEO need to be considered together.
Append offset-continuity checks alone would not detect this case because the divergent records occupy the same offsets.
### Related work
Tracked under #673.
This issue documents a reproducible WAL divergence during a normal,
clean ISR leader migration: a write can be acknowledged while ISR
replicas retain different records at the same offsets.
Related implementation work: #778, #788, and #789.
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with ReassignmentLeaderElection and the replication protocol, then reproduce the three-replica sequence described in the issue. Trace the leader change, follower HW, and local LEO through the reassignment; done means a clean migration establishes a common log prefix before an acks=-1 write is acknowledged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100