lyft / lyft/flinkk8soperator

Update to FlinkApplication in ClusterStarting phase blocks transition to Running phase

Open
#179 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
581
Forks
153
PR merge metrics
No merged PRs in 30d

Description

When a FlinkApplication custom resource is added, the controller of the flink-operator creates job- and taskmanager deployments. These are labelled with a hash value `flink-hash`, which is computed from the FlinkApplication, including all its annotations and labels.

During the lifetime of a FlinkApplication such annotations are sometimes added by other operators. A typical example is the helm-operator (https://github.com/fluxcd/helm-operator). When a FlinkApplication is created by the helm-operator on the basis of a HelmRelease referencing a Helm chart, an annotation `helm.fluxcd.io/antecedent` is added to the FlinkApplication shortly *after* its creation.

If the FlinkApplication is already in its `Running` phase, this leads to an update of the Flink cluster, i.e., the Flink cluster is recreated. This seems to be generally fine, but might be unnecessary when the change to the FlinkApplication does not change the properties of the Flink cluster itself.

However, when the update to the FlinkApplication happens while it is still in the `ClusterStarting` phase, the hash value of the FlinkApplication changes due to the update. As a consequence, the deployments for jobmanager and taskmanagers can not be found as they are still labelled with the original hash value. Therefore, the method IsClusterReady of the controller always returns false, and the FlinkApplication never leaves the `ClusterStarting` phase. See https://github.com/fluxcd/helm-operator/issues/243 .

Maybe an approach would be to compute the hash value not on the basis of the whole FlinkApplication resource, but from the values that actually should update the cluster.

A related problem is that all annotations of the FlinkApplication are propagated to the jobmanager and taskmanager deployments. For annotations like the one of the helm-operator mentioned above, this is not desirable as the annotation is used to identify resources that are explicitly managed by the helm-operator.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the controller's IsClusterReady method and trace how the flink-hash is computed from FlinkApplication metadata and propagated to jobmanager and taskmanager deployments. Reproduce an annotation update during ClusterStarting, then ensure the cluster can reach Running without losing its deployments and that unrelated annotations are not propagated.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.