More dynamic pod_mutation_hook, allowing access to task context and inputs
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 483
Description
### Description
We have some use cases which require fairly complex and dynamic configuration of Kubernets pods. For example configuring the nodeSelector VM size based on task inputs. Also juggling of Volumes, and in particular persistent volume claims. Imagine tasks allocating and preparing volumes which get passed on as input to other tasks.
The mechanism for this today in airflow is
* `pod_template_file` - static and global yaml configuration affecting all tasks
* `pod_mutation_hook` - Sortof dynamic, can run code and has full access to V1Pod object, but is not aware of which Dag, DagRun, Task or TaskInstance is about to be run, can not access the `get_current_context`.
* `executor_config.pod_override` - Static per task configuration. Limited to what `pod_generator.reconcile_xxx` supports, which is a bit limited and can have a some unexpected behavior at times. Not possible to read any DagRun or TaskInstance inputs.
Would it be possible to add a new more rich pod mutation function with following additions:
* Without the reconcile-limitations. (like pod_mutation_hook)
* Aware which dag and task is currently executing. (like pod_override)
* Can read runtime information from the DagRun and TaskInstance. Most importantly `dag_run.conf` and task inputs, eg XCcom return_value from upstream tasks, or perhaps even any xcom value.
### Use case/motivation
_No response_
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Start by tracing the existing pod_mutation_hook, pod_template_file, executor_config.pod_override, and pod_generator.reconcile_xxx entry points. Review how get_current_context exposes DagRun and TaskInstance data, including dag_run.conf and XCom, and locate tests covering pod configuration. Done means a decided, tested approach for dynamic pod mutation with task context and inputs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, python
- Domain
- data-engineering, devops
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100