Current XCom __getitem__ behavior is confusing in task mapping context
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 483
Description
### Apache Airflow version
2.3.3 (latest released)
### What happened
When tasks return dictionary value, it is possible to access the value on the DAG level by using `return_value['key']`. This does not seem to be possible when using the newly added `expand` functionality. It fails to load the DAG with (see example below): `ValueError: cannot map over XCom with custom key 'list' from `
### What you think should happen instead
I think Airflow should behave in a consistent way while using
```
task(return_value['something'])
```
and
```
task.expand(arg=return_value['something'])
```
### How to reproduce
```
from airflow.decorators import dag, task
import pendulum
@dag(
default_args={
'owner': 'airflow',
'start_date': pendulum.datetime(2022, 7, 14, tz="Europe/Prague"),
'depends_on_past': False,
'retries': 0,
},
schedule_interval="* * * * *",
max_active_runs=1,
)
def test():
@task(multiple_outputs=True)
def params():
return {"a": 1, "b": 1, "list": [1, 2, 3]}
parameters = params()
@task
def print_list(list):
return list
@task
def print_elements(element):
return element
#this works
print_list(parameters['list'])
#this also works
print_elements.expand(element=[1,2,3])
#this does not
print_elements.expand(element=parameters['list'])
test_dag = test()
```
### Operating System
Ubuntu 20.0.4
### Versions of Apache Airflow Providers
_No response_
### Deployment
Docker-Compose
### Deployment details
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Start by running the provided DAG example and trace the XCom custom-key handling used by task mapping through the @task and expand entry points shown. Done means dictionary values accessed through return_value['list'] behave consistently in both direct task calls and expand(), with regression coverage for the reproduced failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100