Unbounded sources unable to recover from checkpointMark when withMaxReadTime() is used
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
Observed on v2.22.0
When withMaxReadTime() is used, Beam creates an BoundedReadFromUnboundedSource [1]. The ReadFn class in BoundedReadFromUnboundedSource which is responsible for reading records from source. You can see this class doesnt verify if there is a recoverable checkpoint exist. Instead it always creates Reader with checkpointMark set as null [2].
Reference:
[1] [https://github.com/apache/beam/blob/v2.22.0/sdks/java/core/src/main/java/org/apache/beam/sdk/io/Read.java#L205](https://github.com/apache/beam/blob/v2.22.0/sdks/java/core/src/main/java/org/apache/beam/sdk/io/Read.java#L205)
[2][https://github.com/apache/beam/blob/v2.22.0/sdks/java/core/src/main/java/org/apache/beam/sdk/io/BoundedReadFromUnboundedSource.java#L193](https://github.com/apache/beam/blob/v2.22.0/sdks/java/core/src/main/java/org/apache/beam/sdk/io/BoundedReadFromUnboundedSource.java#L193)
!image-2020-07-15-15-15-12-937.png|width=836,height=346!
Imported from Jira [BEAM-10493](https://issues.apache.org/jira/browse/BEAM-10493). Original Jira may contain additional context.
Reported by: manikolbe.
Contributor guide
Assessment
This issue has not been assessed yet.