apache / apache/airflow

Multiple executors does not work when Airflow is deployed in kubernetes

Open
#71,175 0 comments 0 reactions 0 assignees View on GitHub
area:helm-chart kind:bug needs-triage
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

### Under which category would you file this issue?

Helm chart

### Apache Airflow version

Helm chart 1.22 / Airflow 3.3.0

### What happened and how to reproduce it?

Hello Team,
I have deployed Airflow 3.3.0 in kubernetes and configured the executor Kubernetes executor. Then Airflow was working well and my dags were running in separated pods.

Then I wanted to prepare the use of the new feature called multiple executors.
So, in my dags, I have set executor="KubernetesExecutor" in the task properties.
As a consequence, it returns this error :
UnknownExecutorException: Task 'sync_associations_group.get_data' specifies executor 'KubernetesExecutor', which is not available. Make sure it is listed in your [core] executors configuration, or update the task's executor to use one of the configured executors

I have listed the executor in the executor helm chart property, then I tried in the config.core.executors property, then in config.core.executor property.
Unfortunately it fails every time with the same error.
I have verified the airflow-config configmap and it is correct.

### What you think should happen instead?

It should run my airflow task successfully.

### Operating System

Kubernetes / Official apache/airflow image

### Deployment

Official Apache Airflow Helm Chart

### Apache Airflow Provider(s)

_No response_

### Versions of Apache Airflow Providers

I use the versions from the file https://raw.githubusercontent.com/apache/airflow/constraints-3.3.0/constraints-3.14.txt
and the following providers:
apache-airflow-providers-cncf-kubernetes
apache-airflow-providers-amazon
apache-airflow-providers-opensearch
apache-airflow-providers-http
apache-airflow-providers-common-sql
apache-airflow-providers-postgres

### Official Helm Chart version

1.22.0 (latest released)

### Kubernetes Version

1.33

### Helm Chart configuration

executor: "KubernetesExecutor"
config:
core:
default_timezone: "Europe/Paris"
test_connection: "Enabled"
executors: "KubernetesExecutor"
kubernetes_executor:
worker_container_repository: registry.dso.interieur.rie.gouv.fr/mi-hubassociations/img-airflow-worker
worker_container_tag: *tag_airflow_worker

### Docker Image customizations

_No response_

### Anything else?

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the failure with the provided Official Apache Airflow Helm Chart 1.22.0 configuration and inspect the generated airflow-config ConfigMap, especially the core executors setting. Compare that configuration with the task-level KubernetesExecutor selection; done means a task using multiple executors is accepted and runs successfully in its expected pod.

Written by the indexing model from the issue text.

Assessment

Tech stack
helm, kubernetes, python
Domain
devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.