apache / apache/airflow

Number of queries to Airflow database in "DAG File Processing Stats"

Open
#40,282 7 comments 1 reaction 0 assignees View on GitHub
kind:feature
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Description

This new feature will add a new column to the "DAG File Processing Stats" of DAG processor logs. This column will store information about the number of queries to the Airflow database per DAG.

### Use case/motivation

This new column may be convenient to have it in case of debugging issues related to high load on Airflow database, e.g. typical scenario is when DAG file(s) have a lot of queries to database done on the top level of code and those are executed each time during parsing of these DAG files. One common example is excessive usage of "Variables.get" as top-level statements in DAG files.

Having information about "number of queries to Airflow database" per DAG file may help a lot during debugging issues related to high load on database or issues related to long parsing of the DAG files.

### Related issues

Thread with discussion in the Airflow community: https://lists.apache.org/thread/9j6q2lq521rt5zx46l2dvow2c85sgqwb

### Are you willing to submit a PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start by locating the DAG processor logs and the "DAG File Processing Stats" output, then trace how Variables.get and other top-level DAG queries reach the Airflow database. Done means the stats include a per-DAG database-query count that supports debugging high database load and slow parsing.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering, observability-sre
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.