lyft / lyft/flinkk8soperator

Recovering from total outage / Periodic savepoints

Open
#165 1 comment 4 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
581
Forks
153
PR merge metrics
No merged PRs in 30d

Description

Thank you for effort of making this operator opensource.
I'm evaluating to use it for some pretty critical usecase. Given that, I have HA setup for job managers and high-availability.storageDir is configured to distributed filesystem. One scenario is not quite clear for me: How would it be possible to recover from total flink cluster outage - when all jm and tm's go down. As far as I understand, savepoints are created on upgrades and job cancels, and there is no periodic savepoints. In case of such scenario, application will not have savepoint to recover from. In this case either operator could provide mechanism for periodic savepoints or external process.
There FLIP https://cwiki.apache.org/confluence/display/FLINK/FLIP-10%3A+Unify+Checkpoints+and+Savepoints#
but looks like it's not implemented.
What are your recommendations for this case?
Thank you.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue does not name a file, test, or entry point. Start by reviewing the operator's existing savepoint behavior and the linked Flink FLIP-10 proposal, then determine whether periodic savepoints are supported or need a new design; done should be a documented recovery recommendation or an agreed implementation scope.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
distributed-systems, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.