add a "warning" status for tasks
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 483
Description
### Description
As we can see here https://airflow.apache.org/docs/apache-airflow/1.10.9/concepts.html#id4
The lifecycle of a task is pretty clear, and this lifecycle seems clearly related to the way airflow internally handles the tasks. I propose to have a new status along with Success, failed, shutdown.
This status would be "Warning" or "Success with Warnings" and will act as a success, allowing dependent tasks to run.
### Use case/motivation
In complex ETL pipelines, especially where scrapping is included, you might have a scrapping task at the beginning of the chain that, if failed will prevent all subsequent tasks to run.
You might want to write a really nice scrapper DAG that is smart and detects that there are some issues whit the data scrapped, but that it is ok in some way, or at least is not "THAT" problematic that it requires a failed state and block all other tasks.
In such case, you only have two option: Success or failure. It means that is your task is smart enough to detect that that data scrapped is good enough, but there are problems that should be taken care off, the only option currently is to implement a home made "visualization/notification" system, like sending an email to the inbox so that the person in charge can have a look. Otherwise, it's just binary: ok/nok
It would be wonderful to have a way to set the DAG to this new status so that it can clearly be reflected into the dashboard.
I understand that currently the statuses reflect the way the tasks are handled by airflow and is completely agnostic of what is done inside the task. but in some way, the failed status is already aware that something went wrong into the task.
It would be even better if the task could automatically put itself in status warning when it detects the use of logger.info() or logger.error()
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Start by reviewing the task lifecycle described in the linked Airflow concepts documentation and map how task states affect dependent tasks and dashboard display. Define the warning state's semantics, including whether it behaves as success for dependencies and how it is surfaced, before determining the affected implementation and tests; done means the behavior is specified consistently rather than relying on logger calls.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100