apache / apache/hudi

Add HoodieReadClient with java

Open
#15,129 0 comments 0 reactions 0 assignees View on GitHub
area:reader from-jira priority:critical type:devtask
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

We might need a hoodie read client in java similar to the one we have for spark. 

[Apache Pulsar|https://github.com/apache/pulsar] is doing integration with Hudi, and take Hudi as tiered storage to offload topic cold data into Hudi. When consumers fetch cold data from topic, Pulsar broker will locate the target data is stored in Pulsar or not. If the target data stored in tiered storage (Hudi), Pulsar broker will fetch data from Hudi by Java API, and package them into Pulsar format and dispatch to consumer side.

However, we found current Hudi implementation doesn't support read Hudi table records by Java API, and we couldn't read the target data out from Hudi into Pulsar Broker, which will block the Pulsar & Hudi integration.
h3. What we need
# We need Hudi to support reading records by Java API
# We need Hudi to support read records out which keep the writer order, or support order by specific fields.

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-3892
- Type: Task
- Epic: https://issues.apache.org/jira/browse/HUDI-5197

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by examining the existing Spark read client and the Hudi table-reading APIs. Define the Java API needed by the Pulsar broker, including reading records in writer order or by specified fields. Done means Java callers can read the target Hudi records and the ordering requirement is supported.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.