apache / apache/airflow

Metrics - sla_missed

Open
#32,162 4 comments 0 reactions 0 assignees View on GitHub
_eol_affected_version:2.6 area:core area:metrics kind:bug
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 7h
Merged PRs (30d)
484

Description

### Apache Airflow version

2.6.2

### What happened

The counter `sla_missed` gets emitted in StatsD but not in Otel.

I am submitting this as an Issue since I will be a little distracted for the next bit and figured someone may be able to have a look in the meantime. Please do not assign it to me, I'll get it when I can is nobody else does.

### What you think should happen instead

Behavior should be consistent.

### How to reproduce

To reproduce, you can run Breeze with the statsd or the otel integration (for example breeze start-airflow --integration otel) and run the following DAG, then open the [OTel](http://localhost:28889/metrics) or [StatsD](http://localhost:29102/metrics) raw data view to verify.

```
import time
from datetime import timedelta

from airflow import DAG
from airflow.decorators import task
from airflow.utils.timezone import datetime

def sla_callback(dag, task_list, blocking_task_list, slas, blocking_tis):
print(
"The callback arguments are: ",
{
"dag": dag,
"task_list": task_list,
"blocking_task_list": blocking_task_list,
"slas": slas,
"blocking_tis": blocking_tis,
},
)

@task(sla=timedelta(seconds=10))
def sleep_20():
"""Sleep for 20 seconds"""
time.sleep(20)

@task
def sleep_30():
"""Sleep for 30 seconds"""
time.sleep(30)

with DAG(
dag_id='fail_S_L_A',
start_date=datetime(2021, 1, 1),
schedule="*/2 * * * *",
catchup=False,
sla_miss_callback=sla_callback,
) as dag:

sleep_20() >> sleep_30()
```

You will see `airflow_sla_missed` in the StatsD output but not in the OTel output.

### Operating System

ubuntu

### Versions of Apache Airflow Providers

_No response_

### Deployment

Docker-Compose

### Deployment details

_No response_

### Anything else

_No response_

### Are you willing to submit PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start by running the supplied DAG with `breeze start-airflow --integration otel`, then compare the raw StatsD and OTel views at the listed endpoints. Trace where `airflow_sla_missed` is emitted in each integration and identify the missing OTel path. Done means the `sla_missed` counter appears consistently in both outputs.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
observability-sre
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.