Audit task failed-deps for later investigation
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 483
Description
### Body
Currently Airflow offers [failed-deps](https://airflow.apache.org/docs/apache-airflow/stable/cli-and-env-variables-ref.html#failed-deps) to investigate why task isn't being scheduled. This is very helpful tool however it works only in real time according to the current entries in the metadb. Investigating past anomalies isn't supported.
Sometimes scheduling problems are "solved" on their own. It could be that pool is overcrowded or concurrency has been reached but eventually stress is reduced and tasks are scheduled, thus when you notice it and want to investigate why there was a delay to begin with your capabilities are limited as there could be many reasons.
**The needed solution:**
We should investigate the option to audit the `failed-deps` information or alternatively offer an easy way to export this information in real time to an external audit storage for later investigation.
### Committer
- [X] I acknowledge that I am a maintainer/committer of the Apache Airflow project.
Contributor guide
Research direction
Start with the failed-deps CLI documentation and its current real-time behavior. Investigate whether historical failed-dependency information should be audited or exported to external storage, then define a concrete approach and completion criteria for later investigation of scheduling delays.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cli, data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100