apache / apache/airflow

Dynamic executor_config - templating

Open
#57,898 2 comments 1 reaction 0 assignees View on GitHub
kind:feature needs-triage
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Description

I would like to use templating inside executor_config .

### Use case/motivation

I am using PythonVirtualenvOperator with AwsBatchExecutor which runs the task inside an AWS Batch job.

I would like to dynamiclly tag the job with the dag_id and run_id also set the jobName dynamiclly by runtime params. I need this in order to connect the Airflow task and AWS Batch job, for debugging and looking at other metrics, like cpu and mem consumption.

Actually event more important is the fact that AWS Batch support setting the mem and cpus when submitting a job. This also can't be dynamic and set using params at runtime.

### Related issues

https://github.com/apache/airflow/discussions/57850

### Are you willing to submit a PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start with the PythonVirtualenvOperator and AwsBatchExecutor paths, then review the related discussion at github.com/apache/airflow/discussions/57850 for context. Determine how executor_config is passed to the AWS Batch submission and define completion as allowing runtime templating for tags, jobName, memory, and CPU settings.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud, data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.