AvroReader should refuse dynamic splits while in the last block
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 205
Description
AvroReader is able to detect when it's in the last block:
https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroSource.java#L728
It could also use this information to avoid wastefully producing dynamic splits starting in the range of the current block.
One way to do this would be to have OffsetRangeTracker have a "claim range" operation: claim range of [a, b) is, in terms of correctness, equivalent to claiming "a" (it checks whether "a" is within the range), but sets the last claimed position to "b" rather than "a", thus protecting more positions from being split away.
Imported from Jira [BEAM-2716](https://issues.apache.org/jira/browse/BEAM-2716). Original Jira may contain additional context.
Reported by: jkff.
Contributor guide
Research direction
Start with AvroSource.java around the linked last-block detection and inspect OffsetRangeTracker. Trace how dynamic splits are produced while reading the final block, then identify the relevant tests or entry points in this area. Done means splits cannot begin within the current last block while existing range-claim correctness is preserved.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100