When TabletServer clean shutdown, client will throw OutOfOrderSequenceException
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 625
- Avg merge
- 3d 14h
- Merged PRs (30d)
- 97
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/alibaba/fluss/issues) and found nothing similar.
### Fluss version
main (development)
### Please describe the bug 🐞
When we attempt to kill multiple TabletServers, the client generates a large number of OutOfOrderSequenceException exceptions:

The root cause of this issue is that the current way of updating `highWatermark` is first updates the follower's `highWatermark` and later updates the leader's `highWatermark` in the next round (refer to #676 ). This approach introduces a problem:
During recovery, the follower's `highWatermark` might exceed the `ackedBatchSequence` by the client. For instance:
1. The follower has written batches with batchSequence 3, 4 and updated its `highWatermark`.
2. Meanwhile, the client has not yet received ackes for batches 3, 4, so it still considers its `ackedBatchSequence ` to be 2.
3. **If the leader and follower crashes at this point,** the client assumes the write operation has failed.
4. The follower is then elected as the new leader.
5. When handling a new request for batch 3, the new leader detects an inconsistency, triggering an OutOfOrderSequenceException.
6. The client identifies this scenario as unrecoverable and cannot retry the request, ultimately causing a failover.
This mismatch between the follower’s writerState and the client’s `ackedBatchSequence` directly leads to the failure.
### Solution
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading the highWatermark update approach referenced in issue #676 and trace how TabletServer recovery uses follower and leader state. Reproduce the clean-shutdown scenario with the writerState, highWatermark, and client ackedBatchSequence described here; done means failover no longer produces unrecoverable OutOfOrderSequenceException errors for those requests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100