TieringCommitOperator: Non-atomic dual commit can expose wrong “latest” snapshot for visibility
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 625
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 119
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
### Description
see comment: https://github.com/apache/fluss/pull/2326#discussion_r2782986266
**Problem**
When tiering commits both a readable snapshot and a committed (tiered) snapshot in the same run, the coordinator may record two lake snapshot metadata entries in one commit. The order of these entries is not guaranteed by snapshot timestamp. “latest” can then show the readable snapshot as latest which will be earlier than latest append snapshots. So, it breaks the semantics of latest snapshot.
**Proposed solution**
- Sort the lake snapshot metadatas list by snapshot timestamp (or equivalent total order).
Treat only the most recent snapshot (last after sorting) as the effective committed snapshot for external visibility (e.g. “latest” snapshot).
- Ensure all consumers of “latest snapshot” use this definition so visibility is consistent.
### Willingness to contribute
- [ ] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the tiering commit path and the lake snapshot metadata consumers, then read the linked discussion in PR 2326. Trace how multiple snapshots are ordered and how “latest” is selected; done means external visibility consistently treats the most recent snapshot as latest when readable and committed snapshots are recorded together.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100