Unbounded concurrent Prometheus queries in DeploymentExecutor._fetch_prometheus_metrics

Description

DeploymentExecutor._fetch_prometheus_metrics() executes one Prometheus metric fetch for every matching autoscaling rule.

The method builds the complete set of _fetch_prometheus_metric() coroutines and passes them to a single asyncio.gather() call without a concurrency bound.

As a result, the number of simultaneously active Prometheus queries grows directly with the number of Prometheus-backed autoscaling rules processed in the reconciliation pass.

I reproduced this behavior with 512 controlled autoscaling rules.

Observed behavior:

Prometheus autoscaling rules: 512
Peak active metric queries: 512
Pending child tasks before release: 512
External service contacted: No

For large rule sets, this can produce a burst of concurrent requests against the configured Prometheus service rather than maintaining a bounded level of query concurrency.

Affected Version

Confirmed on:

backend.ai-manager==26.7.0
Python 3.13.14
ai/backend/manager/sokovan/deployment/executor.py
DeploymentExecutor._fetch_prometheus_metrics()

Other released versions have not been tested.

The current main branch also contains the same unbounded asyncio.gather() pattern. The reproduction itself was performed against version 26.7.0.

Reproduction

The attached PoC executes the released _fetch_prometheus_metrics() implementation with 512 controlled Prometheus autoscaling rules.

The individual metric fetch operation is replaced with a local asynchronous double that records active calls and waits before completing. This allows the number of metric queries admitted concurrently by _fetch_prometheus_metrics() to be measured without contacting a Prometheus server.

Observed result:

input rules: 512
peak active metric queries: 512
pending child tasks before release: 512
fan-out reproduced: yes

All 512 metric fetches become active before they are released.

No external service is contacted during reproduction.

Expected behavior

Prometheus metric collection should maintain a bounded number of concurrent queries rather than allowing query concurrency to grow directly with the number of autoscaling rules.

A semaphore, bounded worker pool, or batched execution strategy could preserve concurrent metric collection while limiting the number of simultaneous requests.

Attachment

[run.py](https://github.com/user-attachments/files/31248818/run.py)

[rules.json](https://github.com/user-attachments/files/31253175/rules.json)

Start in ai/backend/manager/sokovan/deployment/executor.py at DeploymentExecutor._fetch_prometheus_metrics() and inspect its asyncio.gather() call and the _fetch_prometheus_metric() entry point. Run the attached run.py with rules.json to reproduce the fan-out. Done means Prometheus metric collection has a bounded concurrency level instead of activating one query per rule.

Written by the indexing model from the issue text.