apache / apache/paimon

[Bug] Job cannot recover from checkpoint/savepoint if parallelism is changed from 1 to 2

Open
#4,543 5 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Paimon version

0.9.0

### Compute Engine

Flink

### Minimal reproduce step

1. start a job to write paimon append only table in parallelism=1.
2. stop the job.
3. restore the job with checkpoint, and change the job's parallelism=2.
4. error appear, job can not restore from checkpoint.

### What doesn't meet your expectations?

Job can restore from checkpoint/savepoint even if I change the parallelism.

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the failure with Apache Paimon 0.9.0 on Flink: write an append-only table at parallelism 1, stop the job, then restore its checkpoint or savepoint at parallelism 2. Inspect the restore error and the checkpoint state-handling path. Done means the job restores successfully after the parallelism change.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.