executor_config: "limits_memory" option for operators
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 483
Description
**Description**
Currently, if you want to change the kubernetes resources allocated to an operator you need to something like
```
from kubernetes.client import models as k8s
task1 = PythonOperator(
task_id="id1",
python_callable=mycallable,
dag=dag1,
executor_config={
"pod_override": k8s.V1Pod(
spec=k8s.V1PodSpec(
containers=[
k8s.V1Container(
name="base",
resources=k8s.V1ResourceRequirements(
limits={
"memory": "6Gi",
"cpu": "1",
}
)
)
]
)
)
},
```
which works, but I think that is very verbose and for a (in my opinion) common case. I would prefer a more direct syntax like
```
task1 = PythonOperator(
task_id="id1",
python_callable=mycallable,
dag=dag1,
executor_config={
"pod_limits_memory": "6Gi",
"pod_limits_cpu": "1",
}
```
Alternatively, maybe a documentation change to
* https://github.com/apache/airflow/blob/main/docs/apache-airflow/executor/kubernetes.rst
* https://github.com/apache/airflow/blob/main/airflow/example_dags/example_kubernetes_executor_config.py
would make this (IMHO common) task easier
**Use case / motivation**
It think the figuring out the `executor_config` > `pod_override` and the `kubernetes.client.models` api just to increase the memory limit for a tasks was kind of hard. I guess a lot of people would benefit from
* exposing a simpler way to achieve it or
* documenting this specific use case in the kubernetes executor documentation
**Are you willing to submit a PR?**
**Related Issues**
Contributor guide
Research direction
Start with docs/apache-airflow/executor/kubernetes.rst and airflow/example_dags/example_kubernetes_executor_config.py, then review the Kubernetes executor configuration described in the issue. The scope needs an explicit decision between a simpler executor_config option and documentation for the existing pod_override approach. Done means the agreed path clearly supports or explains configuring task memory limits.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, python
- Domain
- infrastructure
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100