Manager: GraphQL request metrics leak time series via unbounded operation_name label
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 15h 13m
- Merged PRs (30d)
- 368
Description
## Objective
GraphQLMetricObserver (src/ai/backend/common/metrics/metric.py) labels the Counter backendai_graphql_request_count and the Histogram backendai_graphql_request_duration_sec with operation_name, which is the client-supplied GraphQL operation name (info.operation.name.value). Every distinct name creates new Counter/Histogram time series that are never removed, so a client sending uniquely-named queries grows manager memory without bound.
## Evidence (live worker, Linux + memray)
- 1000 distinct operation names grew exposed series from 1,296 to 14,296 (+13 series/op).
- Series were never reclaimed while idle; only a process restart clears them.
- Worker RSS 373->421MB; histogram multiprocess mmap 128KB->8MB.
- memray stats top allocators: prometheus_client exposition/multiprocess (scrape cost of the exploded cardinality).
## Fix
- Remove operation_name from the label set of both metrics (schema-bounded field_name/parent_type remain).
- Drop operation_name from observe_request and its two call sites (gql/extensions/metric.py, gql_legacy/schema.py); keep it for the OTEL span only.
## Acceptance Criteria
- operation_name is no longer a Prometheus label.
- GraphQL request metric cardinality is bounded by the schema, not by client input.
JIRA Issue: BA-6802
Contributor guide
Research direction
Read src/ai/backend/common/metrics/metric.py and inspect GraphQLMetricObserver, then follow its two call sites in gql/extensions/metric.py and gql_legacy/schema.py. Confirm that operation_name is absent from both Prometheus label sets and observe_request calls while remaining available for the OTEL span, and verify that cardinality is bounded by schema fields.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- graphql, python
- Domain
- api, observability
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 74/100