Stats interface: support asynchronous/observable gauge metrics
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 472
Description
### Description
`airflow.sdk.observability.stats.Stats` (and the OTel-backed stats logger under the hood) only
exposes synchronous, push-style metric recording: `Stats.gauge(name, value)` records whatever
value the caller passes *at the moment it's called*. There's no way to register a callback that
the metrics SDK invokes on its own export/collection cadence.
This is a real gap for "current state" gauges — e.g. a thread-pool occupancy count, a
connection-pool checkout count, or a boolean saturation flag. With only a synchronous API,
producers have to hand-roll a polling loop (a `while` loop on a timer that reads current state
and calls `Stats.gauge(...)`) to get periodic samples. That has two failure modes in practice:
1. If the value is only pushed on a state *transition* (e.g. "just became saturated") rather than
on a fixed interval, the metric goes silent for as long as the state doesn't change — which
most gauge-consuming backends (Prometheus/OTel scrape-based pipelines included) treat the same
as "no data," defeating the point of an alertable gauge.
2. Even when done correctly with a polling loop, every producer re-implements the same
"read current value on a timer, catch exceptions so a metrics failure doesn't crash the loop"
boilerplate that the OTel spec's `ObservableGauge` instrument (a callback registered once,
invoked by the SDK itself at export time) already solves generically.
OpenTelemetry's Metrics API has supported asynchronous/observable instruments
(`ObservableCounter`, `ObservableUpDownCounter`, `ObservableGauge`) for a long time. Since
Airflow's OTel-backed stats path is already built on the OTel SDK, it seems feasible to expose
an equivalent on the `Stats` façade — e.g. `Stats.gauge(name, callback=...)` or a dedicated
`Stats.observable_gauge(name, callback)` registration API — so callers with a canonical "current
value" getter don't need to own their own timer loop, and the emitted samples are guaranteed
fresh on every collection cycle rather than only on edges.
### Use case (concrete example)
Surfaced from an internal package building on Airflow's `Stats` interface for scheduling-latency
observability: a saturation flag ("are we at `max_active_dagruns` right now?") is currently only
loggable, not gauge-able, without a hand-rolled polling loop, precisely because the only available
primitive is the synchronous push `Stats.gauge()`.
### Use case
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Start with airflow.sdk.observability.stats.Stats and the OTel-backed stats logger described in the issue, then compare their current synchronous gauge behavior with OpenTelemetry's ObservableGauge model. The work is complete when callers can register a current-value callback that is invoked during collection, with metric failures handled without requiring producer polling loops.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100