Dynamic executor_config - templating
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 483
Description
### Description
I would like to use templating inside executor_config .
### Use case/motivation
I am using PythonVirtualenvOperator with AwsBatchExecutor which runs the task inside an AWS Batch job.
I would like to dynamiclly tag the job with the dag_id and run_id also set the jobName dynamiclly by runtime params. I need this in order to connect the Airflow task and AWS Batch job, for debugging and looking at other metrics, like cpu and mem consumption.
Actually event more important is the fact that AWS Batch support setting the mem and cpus when submitting a job. This also can't be dynamic and set using params at runtime.
### Related issues
https://github.com/apache/airflow/discussions/57850
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Start with the PythonVirtualenvOperator and AwsBatchExecutor paths, then review the related discussion at github.com/apache/airflow/discussions/57850 for context. Determine how executor_config is passed to the AWS Batch submission and define completion as allowing runtime templating for tags, jobName, memory, and CPU settings.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cloud, data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100