calculate_desired_replicas admits every deployment calculation concurrently
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 15h 13m
- Merged PRs (30d)
- 368
Description
**Description**
`DeploymentExecutor.calculate_desired_replicas` in `backend.ai-manager` 26.7.0 passes the complete collection of per-deployment replica calculations to `asyncio.gather` at `ai/backend/manager/sokovan/deployment/executor.py:328`. Every calculation is scheduled immediately, so concurrency grows with the number of deployments rather than being controlled by a local bound.
I reproduced the admission behavior against the exact gather call expression extracted from the released 26.7.0 source using 64 finite local awaitables.
**Observed Behavior**
Before the controlled awaitables were released:
- Input items: 64
- Tasks admitted: 64
- Peak active tasks: 64
- Pending child tasks: 64
- Eager admission reproduced: yes
- External service contacted: no
All per-deployment calculations become active concurrently. Processing a large deployment set can therefore produce proportionally large task fan-out and simultaneous calculation work.
**Affected Version**
Confirmed with:
- `backend.ai-manager==26.7.0`
- `ai/backend/manager/sokovan/deployment/executor.py:328`
- `DeploymentExecutor.calculate_desired_replicas`
- CPython 3.13.14
Other versions have not been tested.
**Reproduction**
The PoC expects the collected `backend_ai_manager-26.7.0.tar.gz` release and the local shared `_site_harness.py` used by this test corpus. Its configuration selects the `asyncio.gather` call at line 328 and supplies 64 awaitables that wait on a local event.
Run `python run.py` from the material directory in that workspace. The harness:
1. Extracts the released source archive.
2. Locates and AST-extracts the exact gather call expression.
3. Supplies 64 controlled awaitables to that expression.
4. Holds their completion while measuring admitted, active, and pending tasks.
5. Releases the event and exits cleanly.
The result shows all 64 tasks admitted and active before release. No external service, GUI, or device is contacted.
**PoC Source Code**
`run.py`:
```python
#!/usr/bin/env python3
from pathlib import Path
import sys
POC_ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(POC_ROOT))
from _site_harness import run
REPO_ROOT = next(
parent
for parent in Path(__file__).resolve().parents
if (parent / "artifacts").is_dir()
)
if __name__ == "__main__":
raise SystemExit(run(Path(__file__).with_name("inputs") / "sites.json", Path(__file__).with_name("result.json"), REPO_ROOT))
```
Contributor guide
Research direction
Start with DeploymentExecutor.calculate_desired_replicas in ai/backend/manager/sokovan/deployment/executor.py at line 328, then run the supplied run.py reproduction. Trace how the per-deployment calculations reach asyncio.gather and identify the local concurrency bound to apply. Done means the reproduction no longer admits all 64 awaitables before release, with existing behavior preserved.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, performance
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 65/100