apache / apache/airflow

Improve Airflow's debugging story

Open
#40,975 25 comments 3 reactions 3 assignees Claimed by @omkar-foss View on GitHub
airflow3.0:candidate area:logging kind:feature
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

## Summary
As we prepare for the release of Airflow 3.0, one of the key areas that need significant enhancement is the debugging experience.

## Current Challenges
- **Insufficient Logging**: Logs are often fragmented, in some cases overly verbose or non-existent and lack sufficient detail to easily trace issues. We should do an audit of the existing logs.
- **Complex Tracebacks**: Debugging stack traces can be difficult due to the complex nature of DAG (Directed Acyclic Graph) execution and requires a full-running Airflow. Airflow's `dag.test` and `task.test` does a good job already but we should see if we can do even better.
- **Error Handling**: Current error messages are not always informative or actionable, making it hard to understand the root cause of failures. We should do an audit of the existing errors.
- **Tooling Integration**: Lack of integration with modern debugging and observability tools hinders the debugging process. Can we create a listing tool or some capabilities in the Airflow CLI that catches obvious errors? `airflow dags parse` does a job at it, worth checking if it is sufficient or not.

Whoever takes on this task should conduct a user research on the mailing list, Slack, Meetup or Airflow Summit to identify other common debugging problems that can be fixed.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.