apache / apache/hudi

[SUPPORT] When the downstream tasks read the logfile, use the startoffset of each logfile recorded in the deltacommit metadata to read the logfile

Open
#8,425 4 comments 0 reactions 0 assignees View on GitHub
priority:medium type:feature
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

Now, we save the startOffset written to each logfile for this deltacommit. Can we use this data to reduce read amplification when downstream tasks read logfiles?

Contributor guide

No contributing guide indexed for this repository

Research direction

The issue names no files, tests, or entry points. Start by tracing how deltacommit metadata records each logfile's startOffset and how downstream tasks read logfiles. Done means downstream reads use the recorded offsets and reduce read amplification, with coverage for the resulting behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.