Metrics - airflow_local_task
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 484
Description
### Apache Airflow version
2.6.2
### What happened
It is my understanding that the `airflow_local_task_*` metric is only called when the task is explicitly called using airflow cli command, locally. It is getting triggered and reported by StatsD for DAGs run in the UI, but not getting reported by OTel.
I'm not sure if the bugfix here is that this should not be getting emitted in either one when triggered via GUI, or if we need to figure out why OTel is not getting it.
I am submitting this as an Issue since I will be a little distracted for the next bit and figured someone may be able to have a look in the meantime. Please do not assign it to me, I'll get it when I can is nobody else does.
### What you think should happen instead
Behavior should be consistent.
### How to reproduce
To reproduce, you can run Breeze with the statsd or the otel integration (for example breeze start-airflow --integration otel) and run the following DAG, then open the [OTel](http://localhost:28889/metrics) or [StatsD](http://localhost:29102/metrics) raw data view to verify.
```
from airflow import DAG
from airflow.decorators import task
from airflow.utils.timezone import datetime
@task
def task1():
return 'Hello'
@task
def task2():
return 'World!'
@task
def task3(in1, in2):
print(f'{in1} {in2}')
with DAG(
dag_id='taskflow_demo',
start_date=datetime(2021, 1, 1),
schedule=None,
catchup=False
) as dag:
task3(task1(), task2())
```
You will find the following output in the StatsD logs, but no matching values in the OTel logs
```
# HELP airflow_local_task_job_task_exit Metric autogenerated by statsd_exporter.
# TYPE airflow_local_task_job_task_exit counter
airflow_local_task_job_task_exit 3
# HELP airflow_local_task_job_task_exit_3_taskflow_demo_task2_0 Metric autogenerated by statsd_exporter.
# TYPE airflow_local_task_job_task_exit_3_taskflow_demo_task2_0 counter
airflow_local_task_job_task_exit_3_taskflow_demo_task2_0 1
# HELP airflow_local_task_job_task_exit_4_taskflow_demo_task1_0 Metric autogenerated by statsd_exporter.
# TYPE airflow_local_task_job_task_exit_4_taskflow_demo_task1_0 counter
airflow_local_task_job_task_exit_4_taskflow_demo_task1_0 1
# HELP airflow_local_task_job_task_exit_5_taskflow_demo_task3_0 Metric autogenerated by statsd_exporter.
# TYPE airflow_local_task_job_task_exit_5_taskflow_demo_task3_0 counter
airflow_local_task_job_task_exit_5_taskflow_demo_task3_0 1
```
### Operating System
ubuntu
### Versions of Apache Airflow Providers
_No response_
### Deployment
Docker-Compose
### Deployment details
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Start with the provided TaskFlow DAG and run it through Breeze using the statsd and OTel integrations, then compare the raw metrics at the two documented endpoints. Determine the intended consistent behavior for airflow_local_task_* and verify that the corresponding output is present or absent in both integrations.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker-compose, python
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100