apache / apache/airflow

Allow taskflow tasks to simultaneously return an XCOM and mark themselves as failed.

Open
#69,797 1 comment 0 reactions 0 assignees View on GitHub
kind:feature needs-triage
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

### Description

Taskflow is designed to abstract away a lot of the base Airflow concepts like xcom, hence the task return values are converted to xcom automatically. Similarly you don't have to explicitly mark a task as failed, you can just raise an exception.

However there is one gap in this abstraction, namely when you want to do both at once. There is nothing in airflow that prevents a task from pushing an xcom and also failing, but taskflow provides no abstraction for this. At present you would have to explicitly use `push_xcom` and then `raise`. Therefore I suggest a new interface for this, which is a new class called something like `AirflowResultException`, which takes a value that will get pushed to xcom, but the task will still be failed. For example:

```python
class AirflowResultException(AirflowException):
def __init__(self, value: Any):
...

@task
def my_task(x: bool):
if x:
return "foo" # "foo" is pushed as an xcom, and the task status is set to succeeded
else:
raise AirflowResultException("bar") # "bar" is pushed as an xcom, and the task status is set to failed
```

### Use case/motivation

_No response_

### Related issues

_No response_

### Are you willing to submit a PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start by tracing how the @task interface converts return values to XCom and how AirflowException or an explicit push_xcom followed by raise marks a task failed. Done means a task can raise the proposed result exception, preserve its value as XCom, and still finish in the failed state, with coverage for both return and exception paths.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.