AvroSource should update progress within blocks
Open
core
improvement
P3
python
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
Currently Python AvroSource only updates progress at block boundaries.
Java SDK updates this within blocks as well: https://github.com/apache/incubator-beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/BlockBasedSource.java#L208
Imported from Jira [BEAM-663](https://issues.apache.org/jira/browse/BEAM-663). Original Jira may contain additional context.
Reported by: chamikara.
Contributor guide
Research direction
Compare the Python AvroSource implementation with the linked Java BlockBasedSource reference, starting with how progress is currently updated at block boundaries. Update the Python behavior so progress also changes within blocks, and verify that progress reporting matches the intended Java behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100