Allow usage of Jinja templating inside task `param`
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 484
Description
### Description
Currently, it is not possible to use Jinja templates inside the `params` field of a task.
It is not possible to do the following :
```python
bo = BashOperator(
task_id="bash_task",
bash_command = "do-something.sh {{params.path}}",
params = {"path": "/path/to/thing/{{ ds }}/thing.csv"}
)
```
### Use case/motivation
We have a case using the `PostgresOperator` :
```python
PostgresOperator(
task_id="do_postgres_operation",
sql="my_sql_script.sql",
params={
"schema": "my_schema",
"bucket": "my_bucket"
}
)
```
Say `my_sql_script.sql` is a script that let you copy data from S3 bucket to Postgres. It's pretty common and we would like to reuse it here and there. But in our case, we get the `s3_file_name` from the xcom of the previous task, so to get that filename we need to use `{{ ti.xcom_pull(task_ids='my_previous_task') }}` and we would like to use it in `params` that way :
```python
...
params={
"schema": "my_schema",
"bucket": "my_bucket",
"s3_file_name": "{{ ti.xcom_pull(task_ids='my_previous_task') }}"
}
...
```
But it's not possible, we thus need to hard-code `{{ ti.xcom_pull(task_ids='my_previous_task') }}` inside `my_sql_script.sql` script.
But later in the code, we don't want `s3_file_name` to come from xcom but rather from a python variable, and it's impossible because it's hard coded in the sql template. We have to create an other sql file with the exact same code except for `s3_file_name`.
### Related issues
This issue is related to the problem : https://github.com/apache/airflow/issues/7910
It has been marked as resolved but the 2 PRs associated are closed
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Start by reviewing how BashOperator and PostgresOperator process task params and Jinja templates, then read the related discussion in issue 7910. The change is complete when expressions nested in params, including XCom references, render correctly for both operators without requiring separate SQL or shell templates.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100