apache / apache/paimon

[Support] Checkpoint cannot complete when Flink reads the full data of a table in streaming mode.

Open
#7,541 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Paimon version

V1.3.1

### Compute Engine

Flink V1.20.0

### Minimal reproduce step

When using Flink to read a Paimon primary key table with changelog-producer = none, the read is configured with scan.mode = latest-full and continuous.discovery-interval = 20s.

The core reading code is roughly as follows:

Table table = loadPaimonTable(paimonTableConfig);

// Apply dynamic options for CDC streaming read
Map paimonOptions = buildPaimonOptions(paimonTableConfig);
table = table.copy(paimonOptions);

// Build streaming source that directly produces DataStream
// sourceBounded(false) enables continuous streaming mode
return new FlinkSourceBuilder(table)
.env(env)
.sourceBounded(false)
.build();

The Flink checkpoint interval is 10 minutes, and the checkpoint timeout is 1 hour. It is observed that while processing the full data (snapshot phase), all checkpoints time out and none succeed. Checkpoints only start to succeed after the job begins processing incremental data.

### What doesn't meet your expectations?

Although all data is read successfully, the checkpoints cannot be completed properly. We are concerned that if Flink restarts unexpectedly, the processing progress may not be recoverable.

### Anything else?

1.Please explain why checkpoints cannot be completed successfully while processing the full (snapshot) data.
2.How can I modify the configuration or setup so that Flink can successfully complete checkpoints while processing the full data?

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the reported setup with Paimon 1.3.1, Flink 1.20.0, scan.mode=latest-full, continuous.discovery-interval=20s, and the shown FlinkSourceBuilder entry point. Investigate checkpoint behavior during the snapshot phase versus incremental reads; done means explaining the timeout and identifying a configuration or setup that allows checkpoints to complete during full-data processing.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.