apache / apache/airflow

executor_config: "limits_memory" option for operators

Open
#16,563 2 comments 0 reactions 0 assignees View on GitHub
area:core kind:feature provider:cncf-kubernetes
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

**Description**

Currently, if you want to change the kubernetes resources allocated to an operator you need to something like

```
from kubernetes.client import models as k8s

task1 = PythonOperator(
task_id="id1",
python_callable=mycallable,
dag=dag1,
executor_config={
"pod_override": k8s.V1Pod(
spec=k8s.V1PodSpec(
containers=[
k8s.V1Container(
name="base",
resources=k8s.V1ResourceRequirements(
limits={
"memory": "6Gi",
"cpu": "1",
}
)
)
]
)
)
},
```

which works, but I think that is very verbose and for a (in my opinion) common case. I would prefer a more direct syntax like
```
task1 = PythonOperator(
task_id="id1",
python_callable=mycallable,
dag=dag1,
executor_config={
"pod_limits_memory": "6Gi",
"pod_limits_cpu": "1",
}
```

Alternatively, maybe a documentation change to

* https://github.com/apache/airflow/blob/main/docs/apache-airflow/executor/kubernetes.rst
* https://github.com/apache/airflow/blob/main/airflow/example_dags/example_kubernetes_executor_config.py

would make this (IMHO common) task easier

**Use case / motivation**

It think the figuring out the `executor_config` > `pod_override` and the `kubernetes.client.models` api just to increase the memory limit for a tasks was kind of hard. I guess a lot of people would benefit from
* exposing a simpler way to achieve it or
* documenting this specific use case in the kubernetes executor documentation

**Are you willing to submit a PR?**

**Related Issues**

Contributor guide

Open the contributing guide

Research direction

Start with docs/apache-airflow/executor/kubernetes.rst and airflow/example_dags/example_kubernetes_executor_config.py, then review the Kubernetes executor configuration described in the issue. The scope needs an explicit decision between a simpler executor_config option and documentation for the existing pod_override approach. Done means the agreed path clearly supports or explains configuring task memory limits.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, python
Domain
infrastructure
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.