lablup / lablup/backend.ai

Manager: GraphQL request metrics leak time series via unbounded operation_name label

Open
#12,691 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
15h 13m
Merged PRs (30d)
368

Description

## Objective

GraphQLMetricObserver (src/ai/backend/common/metrics/metric.py) labels the Counter backendai_graphql_request_count and the Histogram backendai_graphql_request_duration_sec with operation_name, which is the client-supplied GraphQL operation name (info.operation.name.value). Every distinct name creates new Counter/Histogram time series that are never removed, so a client sending uniquely-named queries grows manager memory without bound.

## Evidence (live worker, Linux + memray)

- 1000 distinct operation names grew exposed series from 1,296 to 14,296 (+13 series/op).
- Series were never reclaimed while idle; only a process restart clears them.
- Worker RSS 373->421MB; histogram multiprocess mmap 128KB->8MB.
- memray stats top allocators: prometheus_client exposition/multiprocess (scrape cost of the exploded cardinality).

## Fix

- Remove operation_name from the label set of both metrics (schema-bounded field_name/parent_type remain).
- Drop operation_name from observe_request and its two call sites (gql/extensions/metric.py, gql_legacy/schema.py); keep it for the OTEL span only.

## Acceptance Criteria

- operation_name is no longer a Prometheus label.
- GraphQL request metric cardinality is bounded by the schema, not by client input.

JIRA Issue: BA-6802

Contributor guide

Open the contributing guide

Research direction

Read src/ai/backend/common/metrics/metric.py and inspect GraphQLMetricObserver, then follow its two call sites in gql/extensions/metric.py and gql_legacy/schema.py. Confirm that operation_name is absent from both Prometheus label sets and observe_request calls while remaining available for the OTEL span, and verify that cardinality is bounded by schema fields.

Written by the indexing model from the issue text.

Assessment

Tech stack
graphql, python
Domain
api, observability
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.