S3/GCS incr job improvements
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
1.
Current batched commit string is of the type "commit#key", given this if we consider the following sample commit
c1->k1
c1->k2
c1->k3
Lets say if the fetch next end exactly at c1#k3, then every fetch following that we would read entire commit c1 and then ignore.
To solve this we would need another flag inside commit string like
"commit#key#commit_complete_boolean_flag", this "commit_complete_boolean_flag" will help us avoid is sub optimal fetch in case we end up in the above scenario.
2.
[https://github.com/apache/hudi/blob/05ac011316564f97de178b023e8e93ff768c37a4/hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/IncrSourceHelper.java#L183] - QueryRunner.applyOrdering api call might not be needed as after filtering we are anyways order it again. Filtering does not need this ordering, we need to test and remove this ordering call (https://github.com/apache/hudi/pull/9433#discussion_r1291828096)
3.
[https://github.com/apache/hudi/pull/9433#discussion_r1291826591]
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-6687
- Type: Improvement
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/IncrSourceHelper.java around line 183 and review the related discussion in PR 9433. Trace the S3/GCS incremental fetch behavior and its tests, then verify the commit-completion marker avoids rereading a completed commit and that removing QueryRunner.applyOrdering remains correct after filtering.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, google-cloud, java
- Domain
- cloud, data-engineering, stream-processing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100