[Umbrella] Hudi Flink Unified Source Implementation
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
### Feature Description
**What the feature achieves:**
Hudi currently supports reading data via Flink Source Function APIs. However, it lacks a native Flink Source API implementation for consuming Hudi tables as a first-class Flink source. The proposal aims to fill that gap by implementing a Flink Source that adheres to Flink's Source API (introduced in Flink 1.11 and enhanced in 1.13+) to enable efficient, scalable and consistent reading of Hudi dataset in both streaming and batch modes.
**Why this feature is needed:**
Flink FLIP-27 solve several problems/shortcomings in the streaming source interface (SourceFunction) and simultaneously to unify the source interfaces between the batch and streaming APIs. By adopting the new source interface, Flink can read hudi data in batch mode for better processing efficiency of bounded data. It also unblocks Flink Hudi user to use Flink hybrid source to seamless switch reading data from hudi to kafka for back-fill use cases.
### User Experience
**How users will use this feature:**
- Configuration changes needed
- API changes
- Usage examples
### Hudi RFC Requirements
**RFC PR link:** (if applicable)
https://github.com/apache/hudi/tree/master/rfc/rfc-95
**Why RFC is/isn't needed:**
- Does this change public interfaces/APIs? Yes
- Does this change storage format? Yes
- Justification:
The new design will use new Flink Source API for Hudi connectors. It will align the implementation with the Flink
APIs for batch and streaming unification. With the feature, Flink users may use Flink hybrid source to read from Hudi table first then switch to read from Kafka.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-9483
- Type: Epic
- Fix version(s):
- 1.2.0
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.