lyft / lyft/flinkk8soperator

FlinkOperator's state not working as expected

Open
#175 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
581
Forks
153
PR merge metrics
No merged PRs in 30d

Description

I submitted wordcount and other jobs..

I was expecting state flows as follow:

New --> ClusterStarting -> Running

But, When I see in UI, the job as finished already but in operator events it never showed ClusterStarting and Running...
Then, Running to Updating
Updating to ClusterStarting

From Running State, I was expecting it to go to Updating State. But all the logs show is SubmittingJob

On minimum from SubmittingJob to Running should have be captured right?

Logs:

kubectl logs -f -n flinkops flinkops-68b7cc8f7c-8pkzb

time="2020-02-23T18:38:31Z" level=info msg=------------------------------------------------------------------------
time="2020-02-23T18:38:31Z" level=info msg="Using config file: [/etc/flinkops/config/config.yaml]"
time="2020-02-23T18:38:31Z" level=info msg="Config section [logger] updated. Firing updated event."
{"json":{},"level":"info","msg":"Config section [operator] updated. No update handler registered.","ts":"2020-02-23T18:38:31Z"}
{"json":{},"level":"info","msg":"\u0026{ResyncPeriod:30s LimitNamespace: MetricsPrefix:flinkk8soperator ProfilerPort:10254 FlinkIngressURLFormat: UseProxy:false ProxyPort:8001 ContainerNameFormat: Workers:4 BaseBackoffDuration:100ms MaxBackoffDuration:30s MaxErrDuration:5m0s}\n","ts":"2020-02-23T18:38:31Z"}
{"json":{},"level":"info","msg":"Starting profiling server on port [10254]","ts":"2020-02-23T18:38:31Z"}
{"json":{},"level":"info","msg":"Registering Components.","ts":"2020-02-23T18:38:34Z"}
{"json":{},"level":"info","msg":"Adding controllers.","ts":"2020-02-23T18:38:34Z"}
{"json":{},"level":"info","msg":"Starting the Cmd.","ts":"2020-02-23T18:38:34Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":""},"level":"info","msg":"Handling state for application","ts":"2020-02-23T18:52:38Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":""},"level":"info","msg":"Logged Normal event: CreatingCluster: Creating Flink cluster for deploy 7df2a560","ts":"2020-02-23T18:52:38Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"ClusterStarting"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T18:52:38Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"ClusterStarting"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T18:53:04Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"ClusterStarting"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T18:53:34Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"ClusterStarting"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T18:54:04Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"ClusterStarting"},"level":"info","msg":"Flink cluster has started successfully","ts":"2020-02-23T18:54:04Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"Savepointing"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T18:54:04Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T18:54:04Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"No job found for the application","ts":"2020-02-23T18:54:04Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"Logged Normal event: JobSubmitted: Flink job submitted to cluster with id 3f6f38c1fbb45d44a406a8fd92b580af","ts":"2020-02-23T18:54:04Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"warning","msg":"Conflict while updating status","ts":"2020-02-23T18:54:04Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"warning","msg":"Failed to reconcile resource flink/wordcount-operator-example: Operation cannot be fulfilled on flinkapplications.flink.k8s.io \"wordcount-operator-example\": the object has been modified; please apply your changes to the latest version and try again","ts":"2020-02-23T18:54:04Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T18:54:04Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"warning","msg":"Found already-submitted job for application with id 3f6f38c1fbb45d44a406a8fd92b580af","ts":"2020-02-23T18:54:04Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T18:54:04Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T18:54:34Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T18:55:04Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T18:55:34Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T18:56:04Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T18:56:34Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T18:57:04Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T18:57:34Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T18:58:04Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T18:58:34Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T18:59:04Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T18:59:34Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T19:00:04Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T19:00:34Z"}



{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T19:01:04Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T19:01:34Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T19:02:04Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T19:02:34Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T19:03:04Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T19:03:34Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T19:04:04Z"}

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the operator's state-handling and reconciliation path, using the provided kubectl logs and the transitions around ClusterStarting, SubmittingJob, and Running. Investigate the status-update conflict and repeated SubmittingJob handling. Done means the expected state transitions are reflected in operator events and the job no longer remains in SubmittingJob unexpectedly.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.