apache / apache/fluss

The flushKv and updateHighWatermark need to be atomic operation

Open
#513 0 comments 0 reactions 0 assignees View on GitHub
component=log component=server
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/alibaba/fluss/issues) and found nothing similar.

### Fluss version

main (development)

### Please describe the bug 🐞

In issue #179 , we will ensure that `flushKV` occurs before `updateHighWatermark` in `Replica#maybeIncrementLeaderHW()`. The purpose of this is that when we consume the CDC log to trigger querying the Fluss pk table, we can definitely retrieve the corresponding value. This requirement is crucial for ensuring correctness in the delta join feature. However, even though we guarantee that `flushKV` happens first, we cannot ensure that these two operations are atomic, and there may still be visibility discrepancies.

### Solution

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with Replica#maybeIncrementLeaderHW() and the ordering change described in issue #179, then trace flushKV and updateHighWatermark to understand their visibility to CDC-log consumers. Done means the two operations are atomic from the consumer's perspective, so delta-join queries can reliably retrieve the corresponding Fluss primary-key value.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.