apache / apache/airflow

Extra metrics are being sent to DataDog when using Allow list with pattern matching enabled

Closed
#39,400 6 comments 0 reactions 0 assignees View on GitHub
affected_version:main_branch area:core area:metrics kind:bug Stale Bug Report
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 7h
Merged PRs (30d)
484

Description

### Apache Airflow version

main (development)

### If "Other Airflow 2 version" selected, which one?

_No response_

### What happened?

with `AIRFLOW__METRICS__METRICS_USE_PATTERN_MATCH` set to `true` and `AIRFLOW__METRICS__METRICS_ALLOW_LIST` set to `.*get_astronauts.*` , I notice several other metrics being sent to datadog that don't match the pattern:
![image](https://github.com/apache/airflow/assets/20266953/16510682-9e45-48c1-9a95-ce726996cca5)

### What you think should happen instead?

Following metrics shouldn't be showing up on datadog when the allow list is `.*get_astronauts.*`:

1. `airflow.dag.task.duration.*`
2. `airflow.ti.*`
3. `datadog.*` (Not sure if we have any control to block these)

### How to reproduce

env variables:

AIRFLOW__METRICS__STATSD_DATADOG_ENABLED | true
-- | --
AIRFLOW__METRICS__STATSD_DATADOG_TAGS | testNo:C2
DATADOG_API_KEY | ● ● ● ● ● ● ●
AIRFLOW__METRICS__METRICS_USE_PATTERN_MATCH | true
AIRFLOW__METRICS__METRICS_ALLOW_LIST | `.*print_astronaut_craft.*`

Once you update your datadog key here, you will start seeing metrics in datadog-->metrics-->summary . use the tag `testNo:C2` to filter

### Operating System

Debian GNU/Linux 11 (bullseye)

### Versions of Apache Airflow Providers

_No response_

### Deployment

Astronomer

### Deployment details

_No response_

### Anything else?

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Reproduce the issue with METRICS_USE_PATTERN_MATCH=true, the stated METRICS_ALLOW_LIST pattern, and the Datadog settings, then inspect the metrics allow-list and pattern-matching path. Use the testNo:C2 tag to review the resulting metrics in Datadog. Done means only metrics matching the configured pattern appear, with the scope of datadog.* behavior clarified.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
observability-sre
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.