apache / apache/hudi

[Flink] Source V2: reuse consumed BatchRecords and row objects via an object pool

Open
#19,302 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Background

PR #19202 reworked the Flink Source V2 read path so that each `BatchRecords` holds a materialized, bounded minibatch of records instead of a live `ClosableIterator`. As a result, `BatchRecords#recycle()` is now a no-op: the batch owns no live I/O, and the reader function (`AbstractSplitReaderFunction`) owns and closes the iterator, `CdcImageManager` and file-group readers on the split-fetcher thread.

### Follow-up

Raised by @cshuo during review of #19202: explore reusing consumed batches (and their row objects) through an object pool, so `recycle()` can return the buffer and its `RowData` copies to a pool instead of discarding them. This could reduce allocation pressure on the source side when a read produces many small minibatches.

### Scope and constraints

- Source-side only (`hudi-flink` reader path: `AbstractSplitReaderFunction`, `BatchRecords`, `HoodieSourceSplitReader`).
- Must preserve the current copy-on-materialize semantics: each buffered record is a distinct copy (not the reader's reused `RowData`) and its `RowKind` is retained.
- Must preserve the existing offset continuity, checkpoint-resume skipping, global limit push-down, and two-phase split-finish contract.
- Pooled objects must be recycled only after the task thread has finished draining a batch, so a pooled row is never handed back while still referenced downstream.

Link back: #19202

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading PR #19202 and the source-side classes AbstractSplitReaderFunction, BatchRecords, and HoodieSourceSplitReader. Trace materialization, recycle(), offset and checkpoint handling, then verify that pooled batches and RowData copies are returned only after draining while preserving RowKind, limits, split completion, and resume behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
stream-processing
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.