lyft / lyft/flinkk8soperator

Flink Native HA on Kubernetes is not supported

Open
#243 4 comments 3 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
581
Forks
153
PR merge metrics
No merged PRs in 30d

Description

I tried out the [native HA on Kubernetes](https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/ha/kubernetes_ha/) using the operator.

Here is the general synopsis:
- If I start a fresh new deployment, the deployment succeeds and the application comes up perfectly
- If I try to perform a deployment (update) of an existing application, then the deployment fails.

The deployment (update) of an existing application goes through the following:
- New job manager and task manager pods for the new deployment are created
- The old job is canceled on the old pods
- The new job manager tries to start the job and prints "Submitting Job with JobId=<>", but fails repeatedly with "The connection was unexpectedly closed by the client."
- The old job manager eventually starts a job with a new job id
- The new pods are destroyed and the old cluster continues running with the old code

Has anyone been successful in using Native K8s HA with Flink with this FlinkK8sOperator?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the linked Native HA on Kubernetes documentation and reproduce both a fresh deployment and an update of an existing application using the FlinkK8sOperator. Inspect the new and old job manager logs around job submission and the connection closure; done means an update succeeds and the new pods run the new code without reverting to the old cluster.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes
Domain
infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.