apache / apache/hudi

S3/GCS incr job improvements

Open
#16,174 0 comments 0 reactions 0 assignees View on GitHub
from-jira priority:medium type:improvement
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

1.

Current batched commit string is of the type "commit#key", given this if we consider the following sample commit
c1->k1
c1->k2
c1->k3

Lets say if the fetch next end exactly at c1#k3, then every fetch following that we would read entire commit c1 and then ignore. 

To solve this we would need another flag inside commit string like
"commit#key#commit_complete_boolean_flag", this "commit_complete_boolean_flag" will help us avoid is sub optimal fetch in case we end up in the above scenario.

2.

[https://github.com/apache/hudi/blob/05ac011316564f97de178b023e8e93ff768c37a4/hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/IncrSourceHelper.java#L183] - QueryRunner.applyOrdering api call might not be needed as after filtering we are anyways order it again. Filtering does not need this ordering, we need to test and remove this ordering call (https://github.com/apache/hudi/pull/9433#discussion_r1291828096)

 

3.

[https://github.com/apache/hudi/pull/9433#discussion_r1291826591]

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-6687
- Type: Improvement

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/IncrSourceHelper.java around line 183 and review the related discussion in PR 9433. Trace the S3/GCS incremental fetch behavior and its tests, then verify the commit-completion marker avoids rereading a completed commit and that removing QueryRunner.applyOrdering remains correct after filtering.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, google-cloud, java
Domain
cloud, data-engineering, stream-processing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.