apache / apache/airflow

Metrics - Possible race condition?

Open
#32,163 4 comments 0 reactions 0 assignees View on GitHub
_eol_affected_version:2.6 area:core area:metrics kind:bug
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 7h
Merged PRs (30d)
484

Description

### Apache Airflow version

2.6.2

### What happened

There are a handful of "end of action" metrics which are emitted in StatsD but not getting emitted in OTel. I believe the solution is to add flush helper method to the SafeOtelLogger which calls MetricsMeter's [force_flush](https://opentelemetry-python.readthedocs.io/en/latest/sdk/metrics.html#opentelemetry.sdk.metrics.MeterProvider.force_flush) then roughly to add something like

```
if hasattr(Stats, "force_flush"):
Stats.meter.force_flush()
```

where the TaskInstance, DagRun, etc are exiting in order to force those metrics to be emitted rather than waiting for the next OTel scheduled pass to collect them. This theory is not yet tested and may be wrong.

I am submitting this as an Issue since I will be a little distracted for the next bit and figured someone may be able to have a look in the meantime. Please do not assign it to me, I'll get it when I can is nobody else does.

### What you think should happen instead

Behavior should be consistent.

### How to reproduce

To reproduce, you can run Breeze with the statsd or the otel integration (for example breeze start-airflow --integration otel) and run the following DAG, then open the [OTel](http://localhost:28889/metrics) or [StatsD](http://localhost:29102/metrics) raw data view to verify.

```
from airflow import DAG
from airflow.decorators import task
from airflow.utils.timezone import datetime

@task
def task1():
return 'Hello'

@task
def task2():
return 'World!'

@task
def task3(in1, in2):
print(f'{in1} {in2}')

with DAG(
dag_id='taskflow_demo',
start_date=datetime(2021, 1, 1),
schedule=None,
catchup=False
) as dag:

task3(task1(), task2())
```

You will find the following counters are visible in the StatsD logs but not in OTel:

- airflow__end
- airflow_operator_failures_
- airflow_operator_successes_
- airflow_ti_failures
- airflow_ti_successes

This one may also be related possibly: https://github.com/apache/airflow/issues/32162

### Operating System

ubuntu

### Versions of Apache Airflow Providers

_No response_

### Deployment

Docker-Compose

### Deployment details

_No response_

### Anything else

_No response_

### Are you willing to submit PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Reproduce the issue with Breeze using the StatsD and OTel integrations, then compare their raw metric views for the listed counters. Read the SafeOtelLogger and MetricsMeter references in the report, along with the TaskInstance and DagRun exit paths; done means the end-of-action counters appear consistently in both integrations.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.