apache / apache/airflow

KubernetesExecutor Worker Configuration in Airflow 3.0

Open
#56,008 2 comments 0 reactions 0 assignees View on GitHub
area:providers kind:bug needs-triage provider:cncf-kubernetes
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Apache Airflow Provider(s)

cncf-kubernetes

### Versions of Apache Airflow Providers

With Airflow 3.0 and the new Task SDK architecture, the airflow worker command no longer exists, but there's insufficient documentation on how to properly configure worker images for KubernetesExecutor environments using the CNCF Kubernetes provider.

## Environment

- Airflow Version: 3.0.6
- Provider Version: apache-airflow-providers-cncf-kubernetes>=8.1.0
- Python Version: 3.12
- Executor: KubernetesExecutor
- Deployment: Kubernetes
- Operating System: Linux (Docker containers)

## Background
We are migrating from Airflow 2.x to 3.0 and need clarification on the proper way to configure worker containers when using KubernetesExecutor from the CNCF Kubernetes provider, especially given the new Task SDK and API server architecture changes in Airflow 3.0.

## Breaking Change: airflow worker Command Removed in Airflow 3.0

The airflow worker command that existed in Airflow 2.x has been completely removed in Airflow 3.0 due to the new Task SDK architecture (AIP-72). In the previous architecture, workers were long-running processes that connected directly to the metadata database. With Airflow 3.0, the architecture has shifted to a client-server model where workers communicate through a dedicated API server instead of direct database access. For KubernetesExecutor specifically, this means worker pods no longer run persistent worker processes - instead, the KubernetesExecutor directly injects task execution commands (like airflow tasks test) into ephemeral pods that execute individual tasks and terminate. This architectural change improves security by removing direct database access from workers and enables better isolation, but requires updating worker image configurations to remove any references to the obsolete airflow worker command.

“Is apache-airflow-providers-cncf-kubernetes compatible? Do you have an example you could share?”

### Apache Airflow version

3.0.6

### Operating System

Linux

### Deployment

Official Apache Airflow Helm Chart

### Deployment details

_No response_

### What happened

_No response_

### What you think should happen instead

_No response_

### How to reproduce

N/A

### Anything else

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start with the official Apache Airflow Helm Chart configuration and KubernetesExecutor behavior described in the issue, using the Airflow 3.0.6 and CNCF Kubernetes provider >=8.1.0 versions. Done means documenting how to configure worker images for KubernetesExecutor, explaining the removed airflow worker command and providing a compatible example.

Written by the indexing model from the issue text.

Assessment

Tech stack
helm, kubernetes, python
Domain
documentation, infrastructure
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.