apache / apache/airflow

Audit task failed-deps for later investigation

Open
#39,712 2 comments 0 reactions 0 assignees View on GitHub
area:scheduler kind:feature
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Body

Currently Airflow offers [failed-deps](https://airflow.apache.org/docs/apache-airflow/stable/cli-and-env-variables-ref.html#failed-deps) to investigate why task isn't being scheduled. This is very helpful tool however it works only in real time according to the current entries in the metadb. Investigating past anomalies isn't supported.

Sometimes scheduling problems are "solved" on their own. It could be that pool is overcrowded or concurrency has been reached but eventually stress is reduced and tasks are scheduled, thus when you notice it and want to investigate why there was a delay to begin with your capabilities are limited as there could be many reasons.

**The needed solution:**
We should investigate the option to audit the `failed-deps` information or alternatively offer an easy way to export this information in real time to an external audit storage for later investigation.

### Committer

- [X] I acknowledge that I am a maintainer/committer of the Apache Airflow project.

Contributor guide

Open the contributing guide

Research direction

Start with the failed-deps CLI documentation and its current real-time behavior. Investigate whether historical failed-dependency information should be audited or exported to external storage, then define a concrete approach and completion criteria for later investigation of scheduling delays.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli, data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.