apache / apache/airflow

Current XCom __getitem__ behavior is confusing in task mapping context

Open
#25,061 4 comments 0 reactions 0 assignees View on GitHub
area:dynamic-task-mapping kind:documentation
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Apache Airflow version

2.3.3 (latest released)

### What happened

When tasks return dictionary value, it is possible to access the value on the DAG level by using `return_value['key']`. This does not seem to be possible when using the newly added `expand` functionality. It fails to load the DAG with (see example below): `ValueError: cannot map over XCom with custom key 'list' from `

### What you think should happen instead

I think Airflow should behave in a consistent way while using
```
task(return_value['something'])
```
and
```
task.expand(arg=return_value['something'])
```

### How to reproduce
```
from airflow.decorators import dag, task
import pendulum

@dag(
default_args={
'owner': 'airflow',
'start_date': pendulum.datetime(2022, 7, 14, tz="Europe/Prague"),
'depends_on_past': False,
'retries': 0,

},
schedule_interval="* * * * *",
max_active_runs=1,
)
def test():
@task(multiple_outputs=True)
def params():
return {"a": 1, "b": 1, "list": [1, 2, 3]}

parameters = params()

@task
def print_list(list):
return list

@task
def print_elements(element):
return element

#this works
print_list(parameters['list'])
#this also works
print_elements.expand(element=[1,2,3])
#this does not
print_elements.expand(element=parameters['list'])

test_dag = test()
```
### Operating System

Ubuntu 20.0.4

### Versions of Apache Airflow Providers

_No response_

### Deployment

Docker-Compose

### Deployment details

_No response_

### Anything else

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start by running the provided DAG example and trace the XCom custom-key handling used by task mapping through the @task and expand entry points shown. Done means dictionary values accessed through return_value['list'] behave consistently in both direct task calls and expand(), with regression coverage for the reproduced failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.