Improve UnboundedSourceShard structure
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 205
Description
Use the 'OneOf' of "Active Reader" and "Optional", to make it obvious how to obtain and use a reader.
Maintain an ordered list of checkpoints that have not been finalized for a shard. Checkpoints that can be resumed from can also be finalized, but otherwise the two are unrelated. This also improves ease of never multi-finalizing a checkpoint. This gets more complicated with retries, but must be considered with the associated work required to make UnboundedReadEvaluatorFactory handle retries properly.
Imported from Jira [BEAM-1659](https://issues.apache.org/jira/browse/BEAM-1659). Original Jira may contain additional context.
Reported by: tgroh.
Contributor guide
Research direction
Start by tracing UnboundedSourceShard and the callers in UnboundedReadEvaluatorFactory, then review how retries, readers, and checkpoints are currently handled. Done means the shard clearly separates an active reader from an optional checkpoint, maintains ordered unfinalized checkpoints, and avoids multi-finalizing checkpoints while accounting for retries.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, distributed-systems
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100