lablup / lablup/backend.ai

calculate_desired_replicas admits every deployment calculation concurrently

Open
#14,273 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
15h 13m
Merged PRs (30d)
368

Description

**Description**

`DeploymentExecutor.calculate_desired_replicas` in `backend.ai-manager` 26.7.0 passes the complete collection of per-deployment replica calculations to `asyncio.gather` at `ai/backend/manager/sokovan/deployment/executor.py:328`. Every calculation is scheduled immediately, so concurrency grows with the number of deployments rather than being controlled by a local bound.

I reproduced the admission behavior against the exact gather call expression extracted from the released 26.7.0 source using 64 finite local awaitables.

**Observed Behavior**

Before the controlled awaitables were released:

- Input items: 64
- Tasks admitted: 64
- Peak active tasks: 64
- Pending child tasks: 64
- Eager admission reproduced: yes
- External service contacted: no

All per-deployment calculations become active concurrently. Processing a large deployment set can therefore produce proportionally large task fan-out and simultaneous calculation work.

**Affected Version**

Confirmed with:

- `backend.ai-manager==26.7.0`
- `ai/backend/manager/sokovan/deployment/executor.py:328`
- `DeploymentExecutor.calculate_desired_replicas`
- CPython 3.13.14

Other versions have not been tested.

**Reproduction**

The PoC expects the collected `backend_ai_manager-26.7.0.tar.gz` release and the local shared `_site_harness.py` used by this test corpus. Its configuration selects the `asyncio.gather` call at line 328 and supplies 64 awaitables that wait on a local event.

Run `python run.py` from the material directory in that workspace. The harness:

1. Extracts the released source archive.
2. Locates and AST-extracts the exact gather call expression.
3. Supplies 64 controlled awaitables to that expression.
4. Holds their completion while measuring admitted, active, and pending tasks.
5. Releases the event and exits cleanly.

The result shows all 64 tasks admitted and active before release. No external service, GUI, or device is contacted.

**PoC Source Code**

`run.py`:

```python
#!/usr/bin/env python3
from pathlib import Path
import sys

POC_ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(POC_ROOT))
from _site_harness import run

REPO_ROOT = next(
parent
for parent in Path(__file__).resolve().parents
if (parent / "artifacts").is_dir()
)

if __name__ == "__main__":
raise SystemExit(run(Path(__file__).with_name("inputs") / "sites.json", Path(__file__).with_name("result.json"), REPO_ROOT))
```

Contributor guide

Open the contributing guide

Research direction

Start with DeploymentExecutor.calculate_desired_replicas in ai/backend/manager/sokovan/deployment/executor.py at line 328, then run the supplied run.py reproduction. Trace how the per-deployment calculations reach asyncio.gather and identify the local concurrency bound to apply. Done means the reproduction no longer admits all 64 awaitables before release, with existing behavior preserved.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, performance
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
65/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.