apache / apache/beam

UnboundedSourceWrapper checkpoints before starting Readers

Open
#19,255 2 comments 0 reactions 0 assignees View on GitHub
bug flink P3 runners
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

When start has not been called on the Readers of CountingSource, the checkpoint may contain a null value which lets the AvroCoder fail with a NullpointerException.

UnboundedSourceWrapper should not attempt to checkpoint before the readers have been started. See UnboundedSource:

> getCheckpointMark not be called before \{@link #start\} has been called.

Imported from Jira [BEAM-6315](https://issues.apache.org/jira/browse/BEAM-6315). Original Jira may contain additional context.
Reported by: mxm.

Contributor guide

Open the contributing guide

Research direction

Start with UnboundedSourceWrapper and the CountingSource readers, then inspect how checkpointing is triggered before reader start. Verify that checkpointing is skipped until readers have started and that the null checkpoint no longer causes the AvroCoder NullPointerException.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
stream-processing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.