apache / apache/fluss

[spark] Port TreeMap-collapse log buffer to FlussUpsertPartitionReader

Open
#3,320 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.

### Description

FlussUpsertPartitionReader.createLogChangesIterator() materializes the full log tail in mutable.ArrayBuffer and sort before merging. So memory grows with total log records.

LakeSnapshotAndLogSplitScanner.pollLogRecords() already implements the right pattern - TreeMap keyed by PK with eager per-PK collapse on put. So memory grows with distinct PKs.

This is the better pattern and more unified one, so let's do it.

### Willingness to contribute

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with FlussUpsertPartitionReader.createLogChangesIterator() and compare it with LakeSnapshotAndLogSplitScanner.pollLogRecords(), focusing on how the log tail is buffered, sorted, and merged. Done means the reader uses the same TreeMap keyed by primary key with eager per-key collapse, so memory scales with distinct PKs rather than total log records.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
stream-processing
Issue type
Refactor
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
70/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.