apache / apache/beam

[Bug]: KinesisIO.Read can't be reconfigured using PipelineOptions when loading from Flink savepoint

Open
#22,707 0 comments 0 reactions 0 assignees View on GitHub
bug flink io java kinesis P2
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

### What happened?

The underlying UnboundedSource of KinesisIO.Read, KinesisSource, can't be re-configured via PipelineOptions once it's serialized in a Flink checkpoint. Specifically the retry and watermark behaviors, which may need to be adjusted in the lifetime of the application, cannot be modified if loading from a savepoint.

Ideally, the `WatermarkPolicyFactory` and `RateLimitPolicyFactory` would accept the PipelineOptions. The PipelineOptions are already provided to `org.apache.beam.sdk.io.kinesis.KinesisSource#createReader` and should be propagated.

### Issue Priority

Priority: 2

### Issue Component

Component: io-java-kinesis

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.