apache / apache/airflow

`airflow tasks run` cli creates a kubernetesJobWatcher when running task with CeleryKubernetesExecutor

Open
#49,074 3 comments 0 reactions 0 assignees View on GitHub
area:CLI area:core kind:bug needs-triage provider:celery
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Apache Airflow version

Other Airflow 2 version (please specify below)

### If "Other Airflow 2 version" selected, which one?

2.10.3

### What happened?

We use CeleryKubernetesExecutor in our setup.

When we try to run a failed task using `airflow tasks run` cli command from airflow worker pod, the task creates a kubernetesJobWatcher and fails as pod does not have the serviceaccount to watch pods.

The task that is run is supposed to run as a celery task, but it is initializing kubernetes executor related things.

Image

Image

### What you think should happen instead?

The task should run as a celery task and should not initialize kubernetes related things.

### How to reproduce

Run a airflow setup using CeleryKubernetesExecutor.
Run a dag with task that runs as a celery task. Fail the task while its running.
Exec into an airflow worker pod and run the failed task using `airflow tasks run` command
Check the logs of the task in the same airflow worker pod.

### Operating System

Debian GNU/Linux 12 (bookworm)

### Versions of Apache Airflow Providers

apache-airflow-providers-celery==3.8.3
apache-airflow-providers-cncf-kubernetes==10.1.0
apache-airflow-providers-common-compat==1.2.1
apache-airflow-providers-common-io==1.4.2
apache-airflow-providers-common-sql==1.19.0
apache-airflow-providers-fab==1.5.0
apache-airflow-providers-ftp==3.11.1
apache-airflow-providers-http==4.13.2
apache-airflow-providers-imap==3.7.0
apache-airflow-providers-smtp==1.8.0
apache-airflow-providers-sqlite==3.9.0

### Deployment

Other

### Deployment details

_No response_

### Anything else?

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start with the `airflow tasks run` entry point and reproduce the command inside a worker pod using CeleryKubernetesExecutor, as described in the issue. Trace why the failed task initializes the Kubernetes job watcher instead of running as a Celery task. Done means the reproduced task no longer requires Kubernetes pod-watching service-account access.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, python
Domain
backend, cli
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.