apache / apache/airflow

More dynamic pod_mutation_hook, allowing access to task context and inputs

Open
#19,673 5 comments 0 reactions 0 assignees View on GitHub
kind:feature
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Description

We have some use cases which require fairly complex and dynamic configuration of Kubernets pods. For example configuring the nodeSelector VM size based on task inputs. Also juggling of Volumes, and in particular persistent volume claims. Imagine tasks allocating and preparing volumes which get passed on as input to other tasks.

The mechanism for this today in airflow is
* `pod_template_file` - static and global yaml configuration affecting all tasks
* `pod_mutation_hook` - Sortof dynamic, can run code and has full access to V1Pod object, but is not aware of which Dag, DagRun, Task or TaskInstance is about to be run, can not access the `get_current_context`.
* `executor_config.pod_override` - Static per task configuration. Limited to what `pod_generator.reconcile_xxx` supports, which is a bit limited and can have a some unexpected behavior at times. Not possible to read any DagRun or TaskInstance inputs.

Would it be possible to add a new more rich pod mutation function with following additions:
* Without the reconcile-limitations. (like pod_mutation_hook)
* Aware which dag and task is currently executing. (like pod_override)
* Can read runtime information from the DagRun and TaskInstance. Most importantly `dag_run.conf` and task inputs, eg XCcom return_value from upstream tasks, or perhaps even any xcom value.

### Use case/motivation

_No response_

### Related issues

_No response_

### Are you willing to submit a PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start by tracing the existing pod_mutation_hook, pod_template_file, executor_config.pod_override, and pod_generator.reconcile_xxx entry points. Review how get_current_context exposes DagRun and TaskInstance data, including dag_run.conf and XCom, and locate tests covering pod configuration. Done means a decided, tested approach for dynamic pod mutation with task context and inputs.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, python
Domain
data-engineering, devops
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.