support for checkpoint-based updates
- Dominant language
- Go
- Stars
- 581
- Forks
- 153
- PR merge metrics
- No merged PRs in 30d
Description
Our Flink job deploys rely heavily on checkpoints since our savepoints take around 30 -45 minutes to write and read back in on the new job.
It appears that enabling of `savepointDisabled` gets us part of the way there and that there exists mechanisms for relying on checkpoints to recover a failing job.
We set `ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION` from within our job and we'd really like to be able to rely only on checkpoints to update jobs.
The way I envision still supporting checkpoints, say for when we need to change parallelism, would be to submit a new job with `savepointDisabled` disabled such that the next job update would use savepoints.
I'm happy to work on this if a PR would be accepted.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing how the operator handles savepointDisabled, ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION, and job updates. Compare the existing savepoint flow with the requested checkpoint-based recovery behavior, including the proposed parallelism-change case. Done means updates can rely on checkpoints by default while still using savepoints when explicitly enabled.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100