apache / apache/hudi

[Umbrella] Hudi Flink Unified Source Implementation

Open
#17,038 0 comments 0 reactions 2 assignees Claimed by @HuangZhenQiu View on GitHub
engine:flink from-jira priority:high status:triaged type:epic
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Feature Description

**What the feature achieves:**
Hudi currently supports reading data via Flink Source Function APIs. However, it lacks a native Flink Source API implementation for consuming Hudi tables as a first-class Flink source. The proposal aims to fill that gap by implementing a Flink Source that adheres to Flink's Source API (introduced in Flink 1.11 and enhanced in 1.13+) to enable efficient, scalable and consistent reading of Hudi dataset in both streaming and batch modes.

**Why this feature is needed:**

Flink FLIP-27 solve several problems/shortcomings in the streaming source interface (SourceFunction) and simultaneously to unify the source interfaces between the batch and streaming APIs. By adopting the new source interface, Flink can read hudi data in batch mode for better processing efficiency of bounded data. It also unblocks Flink Hudi user to use Flink hybrid source to seamless switch reading data from hudi to kafka for back-fill use cases.

### User Experience

**How users will use this feature:**
- Configuration changes needed
- API changes
- Usage examples

### Hudi RFC Requirements

**RFC PR link:** (if applicable)
https://github.com/apache/hudi/tree/master/rfc/rfc-95

**Why RFC is/isn't needed:**
- Does this change public interfaces/APIs? Yes
- Does this change storage format? Yes
- Justification:

The new design will use new Flink Source API for Hudi connectors. It will align the implementation with the Flink
APIs for batch and streaming unification. With the feature, Flink users may use Flink hybrid source to read from Hudi table first then switch to read from Kafka.

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-9483
- Type: Epic
- Fix version(s):
- 1.2.0

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.