apache / apache/fluss

TieringCommitOperator: Non-atomic dual commit can expose wrong “latest” snapshot for visibility

Open
#2,625 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
2d 20h
Merged PRs (30d)
119

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.

### Description

see comment: https://github.com/apache/fluss/pull/2326#discussion_r2782986266

**Problem**
When tiering commits both a readable snapshot and a committed (tiered) snapshot in the same run, the coordinator may record two lake snapshot metadata entries in one commit. The order of these entries is not guaranteed by snapshot timestamp. “latest” can then show the readable snapshot as latest which will be earlier than latest append snapshots. So, it breaks the semantics of latest snapshot.

**Proposed solution**
- Sort the lake snapshot metadatas list by snapshot timestamp (or equivalent total order).
Treat only the most recent snapshot (last after sorting) as the effective committed snapshot for external visibility (e.g. “latest” snapshot).
- Ensure all consumers of “latest snapshot” use this definition so visibility is consistent.

### Willingness to contribute

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the tiering commit path and the lake snapshot metadata consumers, then read the linked discussion in PR 2326. Trace how multiple snapshots are ordered and how “latest” is selected; done means external visibility consistently treats the most recent snapshot as latest when readable and committed snapshots are recorded together.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.