[Flink] Source V2: reuse consumed BatchRecords and row objects via an object pool
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
### Background
PR #19202 reworked the Flink Source V2 read path so that each `BatchRecords` holds a materialized, bounded minibatch of records instead of a live `ClosableIterator`. As a result, `BatchRecords#recycle()` is now a no-op: the batch owns no live I/O, and the reader function (`AbstractSplitReaderFunction`) owns and closes the iterator, `CdcImageManager` and file-group readers on the split-fetcher thread.
### Follow-up
Raised by @cshuo during review of #19202: explore reusing consumed batches (and their row objects) through an object pool, so `recycle()` can return the buffer and its `RowData` copies to a pool instead of discarding them. This could reduce allocation pressure on the source side when a read produces many small minibatches.
### Scope and constraints
- Source-side only (`hudi-flink` reader path: `AbstractSplitReaderFunction`, `BatchRecords`, `HoodieSourceSplitReader`).
- Must preserve the current copy-on-materialize semantics: each buffered record is a distinct copy (not the reader's reused `RowData`) and its `RowKind` is retained.
- Must preserve the existing offset continuity, checkpoint-resume skipping, global limit push-down, and two-phase split-finish contract.
- Pooled objects must be recycled only after the task thread has finished draining a batch, so a pooled row is never handed back while still referenced downstream.
Link back: #19202
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading PR #19202 and the source-side classes AbstractSplitReaderFunction, BatchRecords, and HoodieSourceSplitReader. Trace materialization, recycle(), offset and checkpoint handling, then verify that pooled batches and RowData copies are returned only after draining while preserving RowKind, limits, split completion, and resume behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- stream-processing
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100