apache / apache/airflow

metric: Time after triggering

Open
#19,286 10 comments 0 reactions 0 assignees View on GitHub
area:metrics kind:feature
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Description

Can we have a metric that indicates how long after a dag is triggered its scheduled? I notice that first task scheduling delay doesn't include external triggers but it might be nice to have how long after a dag is triggered it ran to get an idea of things like scheduler failures, full queues etc.

### Use case/motivation

We currently have a critical pipeline and a set of user focused dags that are triggered from the critical pipeline. We would like to have an idea of how fast the scheduler picks up and runs those dags. This is to get a better idea of scheduler responsiveness and also figure out if we need to increase the size of our environment, split environments

### Related issues

_No response_

### Are you willing to submit a PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start by locating Airflow's existing scheduler timing and metric instrumentation for task or DAG scheduling, then review how externally triggered DAG runs are recorded and picked up. Define the trigger-to-run interval and identify the relevant metric tests before implementing the new measurement. Done means the metric reflects externally triggered DAG responsiveness and is covered by tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.