[SUPPORT] When the downstream tasks read the logfile, use the startoffset of each logfile recorded in the deltacommit metadata to read the logfile
Open
priority:medium
type:feature
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
Now, we save the startOffset written to each logfile for this deltacommit. Can we use this data to reduce read amplification when downstream tasks read logfiles?
Contributor guide
No contributing guide indexed for this repository
Research direction
The issue names no files, tests, or entry points. Start by tracing how deltacommit metadata records each logfile's startOffset and how downstream tasks read logfiles. Done means downstream reads use the recorded offsets and reduce read amplification, with coverage for the resulting behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100