lyft / lyft/flinkk8soperator

Failure deployment doesn't change the state in flink operator

Open
#257 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
581
Forks
153
PR merge metrics
No merged PRs in 30d

Description

According to the states, if dual mode, either failure in `ClusterStarting` or `SubmittingJob` will leads to the `RollingBackJob` mode. However, when I tested, I noticed that
1. In `ClusterStaring` state, the deployment failed (usually due to the image not exists or the sidecar is not injected properly that we are missing some packages), the flink app will stuck in `ClusterStarting` and not really goes into the `RollingBackJob` state.
2. In `SubmittingJob`, sometime bad beam python code written and then it will also stuck in `SubmittingJob` state, and I can see that it keeps trying to resubmit the job (and showing error log in flink operator) instead of changing to `RollingBackJob` state as mentioned in the [doc](https://github.com/lyft/flinkk8soperator/blob/master/docs/state_machine.md)

Wondering if I am missing some configuration which leads to this issue. Thanks!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with docs/state_machine.md and compare the documented ClusterStarting and SubmittingJob failure transitions with the observed behavior. Reproduce a failed deployment using the image or sidecar case, then a bad Beam Python submission, and trace the operator logs and state changes. Done means both failures enter RollingBackJob instead of remaining in their current states or repeatedly resubmitting.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.