[spark] Port TreeMap-collapse log buffer to FlussUpsertPartitionReader
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 625
- Avg merge
- 3d 14h
- Merged PRs (30d)
- 97
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
### Description
FlussUpsertPartitionReader.createLogChangesIterator() materializes the full log tail in mutable.ArrayBuffer and sort before merging. So memory grows with total log records.
LakeSnapshotAndLogSplitScanner.pollLogRecords() already implements the right pattern - TreeMap keyed by PK with eager per-PK collapse on put. So memory grows with distinct PKs.
This is the better pattern and more unified one, so let's do it.
### Willingness to contribute
- [x] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with FlussUpsertPartitionReader.createLogChangesIterator() and compare it with LakeSnapshotAndLogSplitScanner.pollLogRecords(), focusing on how the log tail is buffered, sorted, and merged. Done means the reader uses the same TreeMap keyed by primary key with eager per-key collapse, so memory scales with distinct PKs rather than total log records.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- stream-processing
- Issue type
- Refactor
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 70/100