open-telemetry / open-telemetry/opentelemetry-python-contrib
Flask instrumentation: http.server.active_requests gauge leaks on exception (missing try/finally)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 1.1k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 16
Description
Description
The Flask instrumentation's _rewrapped_app function increments active_requests_counter before calling wsgi_app() but the decrement is not wrapped in try/finally. If an exception propagates out of wsgi_app(), the counter is never decremented, causing a permanent gauge leak.
This is particularly impactful when using this metric for Kubernetes HPA (Horizontal Pod Autoscaler) — leaked counters cause HPA to see phantom load and refuse to scale down.
Location
Lines 351–425 in _rewrapped_app:
active_requests_counter.add(1, active_requests_count_attrs) # L351 - increment
# ...
result = wsgi_app(wrapped_app_environ, _start_response) # L398 - if this raises...
# ...duration recording (L400-424)...
active_requests_counter.add(-1, active_requests_count_attrs) # L425 - NEVER REACHED
return result
Expected behavior
The decrement should always execute, regardless of whether wsgi_app() raises an exception.
The WSGI instrumentation already has the correct pattern
try:
with trace.use_span(span):
iterable = self.wsgi(environ, start_response)
return _end_span_after_iterating(iterable, span, token)
except Exception as ex:
raise
finally:
self.active_requests_counter.add(-1, active_requests_count_attrs) # always runs
Suggested fix
Wrap lines 398–425 in try/finally:
active_requests_counter.add(1, active_requests_count_attrs)
request_route = None
# ...
try:
result = wsgi_app(wrapped_app_environ, _start_response)
# ...duration histogram recording...
return result
finally:
active_requests_counter.add(-1, active_requests_count_attrs)
How to reproduce
- Run a Flask app with
opentelemetry-instrumentand gunicorn - Send requests that trigger exceptions propagating past Flask's error handlers (e.g., OOM-killed gunicorn workers, or middleware errors)
- Observe
http.server.active_requestsgauge — it increments but never decrements for crashed requests - The gauge stays permanently elevated until the process is restarted
Impact
- HPA scaling: Kubernetes HPA using this metric sees phantom active requests → refuses to scale down → wasted resources
- Monitoring: Dashboards show incorrect active request counts
- Alerting: False positives on active request alerts
Environment
opentelemetry-instrumentation-flask==0.57b0(also confirmed present onmain/ 0.62b0)- gunicorn with
--workers 1 --threads N --worker-class gthread - OTEL SDK 1.36.0, OTLP HTTP exporter, delta temporality
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in instrumentation/opentelemetry-instrumentation-flask/src/opentelemetry/instrumentation/flask/init.py at _rewrapped_app and compare its request handling with the try/finally pattern in the WSGI instrumentation file. Ensure the active_requests gauge is decremented when wsgi_app() raises, while preserving duration recording and normal return behavior; validate using the exception reproduction described in the issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- flask, python
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100