Using .output on non-templated fields
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 483
Description
### Discussed in https://github.com/apache/airflow/discussions/26938
Originally posted by **stephenonethree** October 7, 2022
I just discovered the `.output` property functionality that apparently was released in Airflow 2 for classic operators, as a simple way of accessing their output XComs. I think that this is a super useful feature because it would allow simpler connections between tasks than what I have been doing until now.
Until now, I've been explicitly giving a downstream task the task_ids and XCom names that it needs to pull from its upstreams (as hardcoded string parameters). Something like:
```
# pushes XCom named myvar
upstream_task=SomeOperator(task_id='upstream_task')
# this is a custom operator and within execute() I have a line roughly like:
# actual_input = context['task_instance'].xcom_pull(
# dag_id=context['dag'].dag_id, task_ids=upstream_task_id, key=upstream_xcom_name)
this_task = OtherOperator(
task_id='this_task',
upstream_task_id='upstream_task',
upstream_xcom_name='myvar'
)
upstream_task >> this_task
```
With `.output` I could simplify this to:
```
# pushes XCom named myvar
upstream_task=SomeOperator(task_id='upstream_task')
this_task = OtherOperator(
task_id='this_task',
actual_input=upstream_task.output['myvar']
)
upstream_task >> this_task
```
Unfortunately it seems that there is one limitation. On the TaskFlow documentation page (https://airflow.apache.org/docs/apache-airflow/2.4.1/tutorial/taskflow.html#consuming-xcoms-between-decorated-and-traditional-tasks) it says: _Using the .output property as an input to another task is supported only for operator parameters listed as a template_field._
I don't use Jinja templating for very many of my parameters as it's mostly irrelevant for me. So my questions are, is there a technical reason for this limitation? If not, is this limitation something that you are considering dropping in a future Airflow version? I suppose I could just make these fields templated to get around it, but I don't really want to turn on templating if I don't expect to need it, since I suppose it introduces the possibility of incorrect interpolation (though perhaps that's a remote possibility because I don't think most of my variables will include `{{` or `}}`.
Let me know if I should file this as a feature request instead, for now I guess the Ideas category works.
Contributor guide
Research direction
Start with the TaskFlow documentation section on consuming XComs between decorated and traditional tasks, focusing on the `.output` limitation for parameters not listed as `template_field`. Investigate the technical reason for this restriction and whether the requested behavior can be supported; done should establish a clear project decision and corresponding behavior or documentation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100