apache / apache/beam

Improve UnboundedSourceShard structure

Open
#18,122 0 comments 0 reactions 0 assignees View on GitHub
direct good first issue improvement P3 runners
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
205

Description

Use the 'OneOf' of "Active Reader" and "Optional", to make it obvious how to obtain and use a reader.

Maintain an ordered list of checkpoints that have not been finalized for a shard. Checkpoints that can be resumed from can also be finalized, but otherwise the two are unrelated. This also improves ease of never multi-finalizing a checkpoint. This gets more complicated with retries, but must be considered with the associated work required to make UnboundedReadEvaluatorFactory handle retries properly.

Imported from Jira [BEAM-1659](https://issues.apache.org/jira/browse/BEAM-1659). Original Jira may contain additional context.
Reported by: tgroh.

Contributor guide

Open the contributing guide

Research direction

Start by tracing UnboundedSourceShard and the callers in UnboundedReadEvaluatorFactory, then review how retries, readers, and checkpoints are currently handled. Done means the shard clearly separates an active reader from an optional checkpoint, maintains ordered unfinalized checkpoints, and avoids multi-finalizing checkpoints while accounting for retries.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, distributed-systems
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.