apache / apache/airflow

Enable kubernetes_pod_operator to reattach_on_restart when the worker dies

Open
#21,900 6 comments 7 reactions 0 assignees View on GitHub
area:providers good first issue kind:feature provider:cncf-kubernetes
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Description

The `kubernetes_pod_operator` currently has a `reattach_on_restart` parameter that attempts to reattach to running pods instead of creating a new pod in case a scheduler dies while the task is running.

We would like for this feature to also work when the worker dies as well. Currently, a dying worker receives a SIGTERM and triggers the `on_kill` method:
https://github.com/apache/airflow/blob/ace8c6e942ff5554639801468b971915b7c0e9b9/airflow/models/taskinstance.py#L1425

This ends up deleting the pod that was created:
https://github.com/apache/airflow/blob/ace8c6e942ff5554639801468b971915b7c0e9b9/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py#L438

We currently got around this problem by removing the the `on_kill` call upon receiving a SIGTERM and pushing an xcom indicating that the worker was killed. We then enabled retries for the `kubernetes_pod_operator` and modified the [is_eligible_to_retry](https://github.com/apache/airflow/blob/ace8c6e942ff5554639801468b971915b7c0e9b9/airflow/models/taskinstance.py#L1825) function to check for the presence of this xcom and only retry if found, allowing us to retry only when the worker was killed.

Unfortunately, this is not a perfect solution because clearing a task / stopping a task via the UI triggers the same signal handler as when a worker is killed externally. Therefore, with this workaround, stopping the task (via UI) now does not kill the pod, and clearing the task (via UI) causes a reattach when we would ideally like a restart.

### Use case/motivation

Since the pod itself may fail for a valid reason, we don't just want to add more retries. In that situation, it will also not re-attach but start a completely new pod since the original pod would have been cleaned up.

We specifically want the reattaching to happen when the worker dies for infrastructure related reasons. This is useful for instance, during deployment updates in kubernetes. It's currently quite a disruptive process because all the running pods are first killed, and if retries are not enabled (for reasons mentioned above), we have to restart all of them again (and potentially lose all the progress on any expensive operations that were running pre-deployment).

### Related issues

_No response_

### Are you willing to submit a PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start with the SIGTERM handling and on_kill call in airflow/models/taskinstance.py around line 1425, then inspect pod cleanup in airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py around line 438 and the reattach_on_restart behavior. Determine how worker death can be distinguished from UI-driven stopping or clearing. Done means worker loss preserves reattachment while ordinary task termination still cleans up or restarts appropriately.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, python
Domain
backend, infrastructure
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.