apache / apache/airflow

TaskFlow unexpectedly unwinds 'dict'

Open
#27,819 5 comments 0 reactions 0 assignees View on GitHub
AIP-31 area:core kind:feature
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Apache Airflow version

main (development)

### What happened

if using the TaskFlow API and one does the following

```
@task()
def MyFunc(myarg: int) -> dict:
return dict({x: "y", z: "x"})

```

Airflow automatically unwinds `dict` automatically into separate variables `x` and `z` and if you do not want that kind of bahavior you will need to set `multiple_outputs=False`. While this is in the documentation, it is rather non intuitive requiring someone to be aware of the documentation that the `default` of `multiple_outputs` changes depending the return type of your function. Moreover, it is unexpected behavior for people coming from plain python (e.g. someone converting their functions to Airflow tasks) and wanting to do:

```
@task()
def f1(myarg: int) -> dict:
return dict({x: myarg, z: "x"})

@task
def f2(in: dict):
print(in)

a = f1(10)
f2(a)

```

Which would suddenly have issues in unittesting.

cc @josh-fell @BasPH @uranusjr

### What you think should happen instead

Airflow should only unwind dicts if explicitly asked to do so.

### How to reproduce

_No response_

### Operating System

Not relevant

### Versions of Apache Airflow Providers

_No response_

### Deployment

Docker-Compose

### Deployment details

_No response_

### Anything else

_No response_

### Are you willing to submit PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start with the TaskFlow API examples using @task and a function annotated to return dict, then trace how multiple_outputs is inferred from the return annotation. Reproduce the f1/f2 example and verify that dictionaries remain single task outputs unless multiple_outputs is explicitly enabled.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.