FlinkOperator's state not working as expected
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 581
- Forks
- 153
- PR merge metrics
- No merged PRs in 30d
Description
I submitted wordcount and other jobs..
I was expecting state flows as follow:
New --> ClusterStarting -> Running
But, When I see in UI, the job as finished already but in operator events it never showed ClusterStarting and Running...
Then, Running to Updating
Updating to ClusterStarting
From Running State, I was expecting it to go to Updating State. But all the logs show is SubmittingJob
On minimum from SubmittingJob to Running should have be captured right?
Logs:
kubectl logs -f -n flinkops flinkops-68b7cc8f7c-8pkzb
time="2020-02-23T18:38:31Z" level=info msg=------------------------------------------------------------------------
time="2020-02-23T18:38:31Z" level=info msg="Using config file: [/etc/flinkops/config/config.yaml]"
time="2020-02-23T18:38:31Z" level=info msg="Config section [logger] updated. Firing updated event."
{"json":{},"level":"info","msg":"Config section [operator] updated. No update handler registered.","ts":"2020-02-23T18:38:31Z"}
{"json":{},"level":"info","msg":"\u0026{ResyncPeriod:30s LimitNamespace: MetricsPrefix:flinkk8soperator ProfilerPort:10254 FlinkIngressURLFormat: UseProxy:false ProxyPort:8001 ContainerNameFormat: Workers:4 BaseBackoffDuration:100ms MaxBackoffDuration:30s MaxErrDuration:5m0s}\n","ts":"2020-02-23T18:38:31Z"}
{"json":{},"level":"info","msg":"Starting profiling server on port [10254]","ts":"2020-02-23T18:38:31Z"}
{"json":{},"level":"info","msg":"Registering Components.","ts":"2020-02-23T18:38:34Z"}
{"json":{},"level":"info","msg":"Adding controllers.","ts":"2020-02-23T18:38:34Z"}
{"json":{},"level":"info","msg":"Starting the Cmd.","ts":"2020-02-23T18:38:34Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":""},"level":"info","msg":"Handling state for application","ts":"2020-02-23T18:52:38Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":""},"level":"info","msg":"Logged Normal event: CreatingCluster: Creating Flink cluster for deploy 7df2a560","ts":"2020-02-23T18:52:38Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"ClusterStarting"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T18:52:38Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"ClusterStarting"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T18:53:04Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"ClusterStarting"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T18:53:34Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"ClusterStarting"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T18:54:04Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"ClusterStarting"},"level":"info","msg":"Flink cluster has started successfully","ts":"2020-02-23T18:54:04Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"Savepointing"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T18:54:04Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T18:54:04Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"No job found for the application","ts":"2020-02-23T18:54:04Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"Logged Normal event: JobSubmitted: Flink job submitted to cluster with id 3f6f38c1fbb45d44a406a8fd92b580af","ts":"2020-02-23T18:54:04Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"warning","msg":"Conflict while updating status","ts":"2020-02-23T18:54:04Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"warning","msg":"Failed to reconcile resource flink/wordcount-operator-example: Operation cannot be fulfilled on flinkapplications.flink.k8s.io \"wordcount-operator-example\": the object has been modified; please apply your changes to the latest version and try again","ts":"2020-02-23T18:54:04Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T18:54:04Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"warning","msg":"Found already-submitted job for application with id 3f6f38c1fbb45d44a406a8fd92b580af","ts":"2020-02-23T18:54:04Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T18:54:04Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T18:54:34Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T18:55:04Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T18:55:34Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T18:56:04Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T18:56:34Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T18:57:04Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T18:57:34Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T18:58:04Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T18:58:34Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T18:59:04Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T18:59:34Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T19:00:04Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T19:00:34Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T19:01:04Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T19:01:34Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T19:02:04Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T19:02:34Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T19:03:04Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T19:03:34Z"}
{"json":{"app_name":"wordcount-operator-example","ns":"flink","phase":"SubmittingJob"},"level":"info","msg":"Handling state for application","ts":"2020-02-23T19:04:04Z"}
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the operator's state-handling and reconciliation path, using the provided kubectl logs and the transitions around ClusterStarting, SubmittingJob, and Running. Investigate the status-update conflict and repeated SubmittingJob handling. Done means the expected state transitions are reflected in operator events and the job no longer remains in SubmittingJob unexpectedly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100