Multiple executors does not work when Airflow is deployed in kubernetes
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 472
Description
### Under which category would you file this issue?
Helm chart
### Apache Airflow version
Helm chart 1.22 / Airflow 3.3.0
### What happened and how to reproduce it?
Hello Team,
I have deployed Airflow 3.3.0 in kubernetes and configured the executor Kubernetes executor. Then Airflow was working well and my dags were running in separated pods.
Then I wanted to prepare the use of the new feature called multiple executors.
So, in my dags, I have set executor="KubernetesExecutor" in the task properties.
As a consequence, it returns this error :
UnknownExecutorException: Task 'sync_associations_group.get_data' specifies executor 'KubernetesExecutor', which is not available. Make sure it is listed in your [core] executors configuration, or update the task's executor to use one of the configured executors
I have listed the executor in the executor helm chart property, then I tried in the config.core.executors property, then in config.core.executor property.
Unfortunately it fails every time with the same error.
I have verified the airflow-config configmap and it is correct.
### What you think should happen instead?
It should run my airflow task successfully.
### Operating System
Kubernetes / Official apache/airflow image
### Deployment
Official Apache Airflow Helm Chart
### Apache Airflow Provider(s)
_No response_
### Versions of Apache Airflow Providers
I use the versions from the file https://raw.githubusercontent.com/apache/airflow/constraints-3.3.0/constraints-3.14.txt
and the following providers:
apache-airflow-providers-cncf-kubernetes
apache-airflow-providers-amazon
apache-airflow-providers-opensearch
apache-airflow-providers-http
apache-airflow-providers-common-sql
apache-airflow-providers-postgres
### Official Helm Chart version
1.22.0 (latest released)
### Kubernetes Version
1.33
### Helm Chart configuration
executor: "KubernetesExecutor"
config:
core:
default_timezone: "Europe/Paris"
test_connection: "Enabled"
executors: "KubernetesExecutor"
kubernetes_executor:
worker_container_repository: registry.dso.interieur.rie.gouv.fr/mi-hubassociations/img-airflow-worker
worker_container_tag: *tag_airflow_worker
### Docker Image customizations
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Start by reproducing the failure with the provided Official Apache Airflow Helm Chart 1.22.0 configuration and inspect the generated airflow-config ConfigMap, especially the core executors setting. Compare that configuration with the task-level KubernetesExecutor selection; done means a task using multiple executors is accepted and runs successfully in its expected pod.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- helm, kubernetes, python
- Domain
- devops
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100