lyft / lyft/flinkk8soperator

Automate handling of dead task managers

Open
#212 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
581
Forks
153
PR merge metrics
No merged PRs in 30d

Description

Occasionally we see task manager JVM process get stuck, with the task manager no longer registered with Flink but the process also not able to exit. This leads to a job recovery crash loop due to insufficient resources (missing task slots). It would be good if the operator could detect lost task managers and delete corresponding pods so that replacement TMs can come up and the application recover.

The manual process is to find the task manager IPs in the pod list that are not registered with Flink (listed in the Flink UI) and then kubectl delete those pods.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the operator's task-manager reconciliation and how it observes Flink registration, using the Flink UI and Kubernetes pod list described in the issue. Verify completion by confirming that an unregistered, stuck task manager pod is detected and deleted so a replacement can start and the application can recover.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
devops, infrastructure
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.