apache / apache/airflow

Using .output on non-templated fields

Open
#27,285 6 comments 0 reactions 0 assignees View on GitHub
good first issue kind:feature
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Discussed in https://github.com/apache/airflow/discussions/26938

Originally posted by **stephenonethree** October 7, 2022
I just discovered the `.output` property functionality that apparently was released in Airflow 2 for classic operators, as a simple way of accessing their output XComs. I think that this is a super useful feature because it would allow simpler connections between tasks than what I have been doing until now.

Until now, I've been explicitly giving a downstream task the task_ids and XCom names that it needs to pull from its upstreams (as hardcoded string parameters). Something like:

```
# pushes XCom named myvar
upstream_task=SomeOperator(task_id='upstream_task')

# this is a custom operator and within execute() I have a line roughly like:
# actual_input = context['task_instance'].xcom_pull(
# dag_id=context['dag'].dag_id, task_ids=upstream_task_id, key=upstream_xcom_name)
this_task = OtherOperator(
task_id='this_task',
upstream_task_id='upstream_task',
upstream_xcom_name='myvar'
)
upstream_task >> this_task
```

With `.output` I could simplify this to:

```
# pushes XCom named myvar
upstream_task=SomeOperator(task_id='upstream_task')

this_task = OtherOperator(
task_id='this_task',
actual_input=upstream_task.output['myvar']
)
upstream_task >> this_task
```

Unfortunately it seems that there is one limitation. On the TaskFlow documentation page (https://airflow.apache.org/docs/apache-airflow/2.4.1/tutorial/taskflow.html#consuming-xcoms-between-decorated-and-traditional-tasks) it says: _Using the .output property as an input to another task is supported only for operator parameters listed as a template_field._

I don't use Jinja templating for very many of my parameters as it's mostly irrelevant for me. So my questions are, is there a technical reason for this limitation? If not, is this limitation something that you are considering dropping in a future Airflow version? I suppose I could just make these fields templated to get around it, but I don't really want to turn on templating if I don't expect to need it, since I suppose it introduces the possibility of incorrect interpolation (though perhaps that's a remote possibility because I don't think most of my variables will include `{{` or `}}`.

Let me know if I should file this as a feature request instead, for now I guess the Ideas category works.

Contributor guide

Open the contributing guide

Research direction

Start with the TaskFlow documentation section on consuming XComs between decorated and traditional tasks, focusing on the `.output` limitation for parameters not listed as `template_field`. Investigate the technical reason for this restriction and whether the requested behavior can be supported; done should establish a clear project decision and corresponding behavior or documentation.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.