apache / apache/airflow

Typing error

Open
#53,640 23 comments 0 reactions 0 assignees View on GitHub
area:async-operators area:core kind:bug
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Apache Airflow version

3.0.3

### If "Other Airflow 2 version" selected, which one?

3.0.2

### What happened?

I wrote a custom deferrable Operator, that uses a custom trigger for deferring. The custom trigger should fail/succeed the task (yielding `TaskFailedEvent` or `TaskSuccessEvent`).

According to the [docstrings](https://github.com/apache/airflow/blob/8fb611667e83ac037ee16722549efedef7350ee7/task-sdk/src/airflow/sdk/bases/operator.py#L1586), I should pass `method_name=None` in this case. However, that leads to a typing issue because `method_name` has type `string `rather than `Optional(string)`

Indeed, the task fails with a validation error:
```
- Top level error: source="task"
ValidationError: 1 validation error for DeferTask
next_method
Input should be a valid string [type=string_type, input_value=None, input_type=NoneType]
```

### What you think should happen instead?

When passing `None` as `next_method`, the task should be failed/succeeded directly from the trigger.

### How to reproduce

Define a custom deferrable Operator, that uses a custom trigger for deferring. The custom trigger should only fail/succeed the task (yielding TaskFailedEvent or TaskSuccessEvent).

Define the custom operator's `execute` function as :
```
def execute(self):
self.defer(
trigger=CustomTrigger(),
method_name=None, # type:ignore
# according to docs:
# If the trigger will end the task instance itself,
# ``method_name`` should be None
# But it's not nullable
)
```

And use the custom operator as a single task in a dag

### Operating System

RedHat Openshift

### Versions of Apache Airflow Providers

_No response_

### Deployment

Official Apache Airflow Helm Chart

### Deployment details

_No response_

### Anything else?

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start in task-sdk/src/airflow/sdk/bases/operator.py, reading the defer method and its method_name docstring. Trace the DeferTask validation reported in the issue and reproduce the custom deferrable operator with method_name=None. Done means the value is accepted and a trigger yielding TaskFailedEvent or TaskSuccessEvent can complete the task directly.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.