apache / apache/airflow

Order of task arguments in task definition causing error when parsing DAG

Open
#56,128 6 comments 0 reactions 0 assignees View on GitHub
affected_version:3.1 area:core area:task-sdk kind:bug needs-triage
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Apache Airflow version

3.1.0

### If "Other Airflow 2 version" selected, which one?

_No response_

### What happened?

Some of my DAGs definitions, that are serialized without errors in 2.11.0, are throwing errors when being serialized in 3.0.0. It seems like some argument names and orderings are not being handled correctly by `@task`.

More info in the reproduction steps.

### What you think should happen instead?

The example DAG in the reproduction steps should be serialized without errors.

### How to reproduce

Create the three following files:

```python
# example_1.py
from airflow.decorators import dag, task

@dag()
def example_1():
@task
def foo(end_date, start_date): ...

foo(None, None)

example_1()
```
```python
# example_2.py
from airflow.decorators import dag, task

@dag()
def example_2():
@task
def foo(start_date, end_date): ...

foo(None, None)

example_2()

```
```shell
# test.sh

docker run \
--rm \
--entrypoint "/bin/bash" \
--volume .:/opt/airflow/dags/ \
"apache/airflow:${1}" \
-c \
"
airflow db migrate > /dev/null 2>&1;
airflow dags report 2>/dev/null;
"
```

Running `./test.sh 2.11.0` successfully serializes both dags, but running `./test.sh 3.0.0` or `./test.sh 3.1.0` only serializes one of the dags:
```
lucas@tiny ~/Desktop/poc % ./test.sh 2.11.0
file | duration | dag_num | task_num | dags
==============+================+=========+==========+==========
/example_1.py | 0:00:00.031625 | 1 | 1 | example_1
/example_2.py | 0:00:00.000719 | 1 | 1 | example_2

lucas@tiny ~/Desktop/poc % ./test.sh 3.0.0
file | duration | dag_num | task_num | dags
==============+================+=========+==========+==========
/example_1.py | 0:00:00.055614 | 1 | 1 | example_1
/example_2.py | 0:00:00.001864 | 0 | 0 |

lucas@tiny ~/Desktop/poc % ./test.sh 3.1.0
file | duration | dag_num | task_num | dags
==============+================+=========+==========+==========
/example_1.py | 0:00:00.015002 | 1 | 1 | example_1
/example_2.py | 0:00:00.007117 | 0 | 0 |

```

This is the traceback of the error I get when serializing the DAG in >=3.0.0:
```
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/bases/decorator.py", line 214, in __init__
signature = signature.replace(parameters=parameters)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/python/lib/python3.12/inspect.py", line 3109, in replace
return type(self)(parameters,
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/python/lib/python3.12/inspect.py", line 3065, in __init__
raise ValueError(msg)
ValueError: non-default argument follows default argument

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/models/dagbag.py", line 405, in parse
loader.exec_module(new_module)
File "", line 999, in exec_module
File "", line 488, in _call_with_frames_removed
File "/opt/airflow/dags/example_2.py", line 12, in
example_2()
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/definitions/dag.py", line 1514, in factory
f(**f_kwargs)
File "/opt/airflow/dags/example_2.py", line 9, in example_2
foo(None, None)
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/bases/decorator.py", line 363, in __call__
op = self.operator_class(
^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/bases/operator.py", line 521, in apply_defaults
result = func(self, **kwargs, default_args=default_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/standard/decorators/python.py", line 58, in __init__
super().__init__(
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/bases/operator.py", line 521, in apply_defaults
result = func(self, **kwargs, default_args=default_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/bases/decorator.py", line 229, in __init__
raise ValueError(message) from err
ValueError:
The function signature broke while assigning defaults to context key parameters.

The decorator is replacing the signature
> foo(start_date, end_date)

with
> foo(start_date=None, end_date)

which isn't valid: non-default argument follows default argument
```

### Operating System

Debian 12

### Versions of Apache Airflow Providers

None

### Deployment

Other

### Deployment details

Not relevant.

### Anything else?

_No response_

### Are you willing to submit PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Reproduce the failure with example_1.py, example_2.py, and test.sh against the listed Airflow versions. Then inspect airflow/sdk/bases/decorator.py around the signature replacement shown in the traceback. Done means both example DAGs serialize successfully on Airflow 3.0.0 and 3.1.0 without the ValueError.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.