Automate handling of dead task managers
- Dominant language
- Go
- Stars
- 581
- Forks
- 153
- PR merge metrics
- No merged PRs in 30d
Description
Occasionally we see task manager JVM process get stuck, with the task manager no longer registered with Flink but the process also not able to exit. This leads to a job recovery crash loop due to insufficient resources (missing task slots). It would be good if the operator could detect lost task managers and delete corresponding pods so that replacement TMs can come up and the application recover.
The manual process is to find the task manager IPs in the pod list that are not registered with Flink (listed in the Flink UI) and then kubectl delete those pods.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing the operator's task-manager reconciliation and how it observes Flink registration, using the Flink UI and Kubernetes pod list described in the issue. Verify completion by confirming that an unregistered, stuck task manager pod is detected and deleted so a replacement can start and the application can recover.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- devops, infrastructure
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100