apache / apache/airflow

dag_processing.last_duration.<dag_file> metric missing

Open
#60,325 3 comments 0 reactions 0 assignees View on GitHub
area:core area:metrics kind:bug needs-triage provider:amazon
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 7h
Merged PRs (30d)
484

Description

### Apache Airflow version

Other Airflow 3 version (please specify below)

### If "Other Airflow 3 version" selected, which one?

3.0.6

### What happened?

Prior to version 3x, Airflow emitted the `dag_processing.last_duration.` metric. While working in MWAA with version 3.0.6, I noticed this metric is no longer available in CloudWatch. I verified MWAA is not excluding any metrics, so I expect the metric to be present.

For a sanity check, I enabled StatsD in Breeze to see if the metric is emitted or not. I don't see the metric in Grafana.

The other python file-level metric (dag_processing.last_run.seconds_ago.) is still emitted and visible in CloudWatch.

The metric is still listed in the [Airflow metrics documentation](https://airflow.apache.org/docs/apache-airflow/3.0.6/administration-and-deployment/logging-monitoring/metrics.html#timers).

### What you think should happen instead?

I expected the metric to be present. I find it useful when identifying python files that may need to be tuned for performance.

### How to reproduce

For MWAA, create an MWAA environment with version 3.0.6 and upload a python file (with dag) to the Amazon S3 bucket. Once parsing takes place, attempt to find the metric in Amazon CloudWatch.

### Operating System

AL2023

### Versions of Apache Airflow Providers

_No response_

### Deployment

Amazon (AWS) MWAA

### Deployment details

This is a standard MWAA deployment. I did not add/modify any of the `[metrics]` config items.

### Anything else?

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start with the Airflow metrics documentation and reproduce the issue by enabling StatsD in Breeze, comparing the missing last_duration metric with dag_processing.last_run.seconds_ago. Trace the metric emission path and verify the result in StatsD or Grafana; done means the metric is emitted again and remains consistent with the documented timers.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
observability
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
39/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.