apache / apache/fluss

When TabletServer clean shutdown, client will throw OutOfOrderSequenceException

Open
#709 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/alibaba/fluss/issues) and found nothing similar.

### Fluss version

main (development)

### Please describe the bug 🐞

When we attempt to kill multiple TabletServers, the client generates a large number of OutOfOrderSequenceException exceptions:

![Image](https://github.com/user-attachments/assets/5b6bd053-9df7-41c1-b902-b6dedfe82d73)

The root cause of this issue is that the current way of updating `highWatermark` is first updates the follower's `highWatermark` and later updates the leader's `highWatermark` in the next round (refer to #676 ). This approach introduces a problem:
During recovery, the follower's `highWatermark` might exceed the `ackedBatchSequence` by the client. For instance:

1. The follower has written batches with batchSequence 3, 4 and updated its `highWatermark`.
2. Meanwhile, the client has not yet received ackes for batches 3, 4, so it still considers its `ackedBatchSequence ` to be 2.
3. **If the leader and follower crashes at this point,** the client assumes the write operation has failed.
4. The follower is then elected as the new leader.
5. When handling a new request for batch 3, the new leader detects an inconsistency, triggering an OutOfOrderSequenceException.
6. The client identifies this scenario as unrecoverable and cannot retry the request, ultimately causing a failover.
This mismatch between the follower’s writerState and the client’s `ackedBatchSequence` directly leads to the failure.

### Solution

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading the highWatermark update approach referenced in issue #676 and trace how TabletServer recovery uses follower and leader state. Reproduce the clean-shutdown scenario with the writerState, highWatermark, and client ackedBatchSequence described here; done means failover no longer produces unrecoverable OutOfOrderSequenceException errors for those requests.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.