apache / apache/airflow

Stats interface: support asynchronous/observable gauge metrics

Open
#72,885 0 comments 0 reactions 0 assignees View on GitHub
kind:feature
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

### Description

`airflow.sdk.observability.stats.Stats` (and the OTel-backed stats logger under the hood) only
exposes synchronous, push-style metric recording: `Stats.gauge(name, value)` records whatever
value the caller passes *at the moment it's called*. There's no way to register a callback that
the metrics SDK invokes on its own export/collection cadence.

This is a real gap for "current state" gauges — e.g. a thread-pool occupancy count, a
connection-pool checkout count, or a boolean saturation flag. With only a synchronous API,
producers have to hand-roll a polling loop (a `while` loop on a timer that reads current state
and calls `Stats.gauge(...)`) to get periodic samples. That has two failure modes in practice:

1. If the value is only pushed on a state *transition* (e.g. "just became saturated") rather than
on a fixed interval, the metric goes silent for as long as the state doesn't change — which
most gauge-consuming backends (Prometheus/OTel scrape-based pipelines included) treat the same
as "no data," defeating the point of an alertable gauge.
2. Even when done correctly with a polling loop, every producer re-implements the same
"read current value on a timer, catch exceptions so a metrics failure doesn't crash the loop"
boilerplate that the OTel spec's `ObservableGauge` instrument (a callback registered once,
invoked by the SDK itself at export time) already solves generically.

OpenTelemetry's Metrics API has supported asynchronous/observable instruments
(`ObservableCounter`, `ObservableUpDownCounter`, `ObservableGauge`) for a long time. Since
Airflow's OTel-backed stats path is already built on the OTel SDK, it seems feasible to expose
an equivalent on the `Stats` façade — e.g. `Stats.gauge(name, callback=...)` or a dedicated
`Stats.observable_gauge(name, callback)` registration API — so callers with a canonical "current
value" getter don't need to own their own timer loop, and the emitted samples are guaranteed
fresh on every collection cycle rather than only on edges.

### Use case (concrete example)

Surfaced from an internal package building on Airflow's `Stats` interface for scheduling-latency
observability: a saturation flag ("are we at `max_active_dagruns` right now?") is currently only
loggable, not gauge-able, without a hand-rolled polling loop, precisely because the only available
primitive is the synchronous push `Stats.gauge()`.

### Use case

_No response_

### Are you willing to submit a PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start with airflow.sdk.observability.stats.Stats and the OTel-backed stats logger described in the issue, then compare their current synchronous gauge behavior with OpenTelemetry's ObservableGauge model. The work is complete when callers can register a current-value callback that is invoked during collection, with metric failures handled without requiring producer polling loops.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.