Autoscaling is silently skipped when step_size crosses min/max_replicas
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 17h 7m
- Merged PRs (30d)
- 358
Description
### What Operating System(s) are you seeing this problem on?
Linux (x86-64)
### Backend.AI version
26.4.10
### Describe the bug
Even when an autoscaling rule's threshold condition is met, if `current_replicas ± step_size` falls
outside the `min_replicas` / `max_replicas` range, the value is **not clamped to the boundary — the
scaling action is cancelled entirely**.
`repositories/deployment/repository.py:1081-1101` treats the bounds as a skip condition rather than a clamp.
```python
new_replica_count = max(0, current_replica_count + scale_direction * rule.action.step_size)
if rule.action.min_replicas is not None and new_replica_count < rule.action.min_replicas:
log.info("AUTOSCALE(...): new count {} below min {}", ...)
continue # ← gives up on scaling
if rule.action.max_replicas is not None and new_replica_count > rule.action.max_replicas:
log.info("AUTOSCALE(...): new count {} above max {}", ...)
continue # ← gives up on scaling
```
As a result, any configuration where `step_size` straddles a boundary creates a lock-up in which
**scaling never happens from any replica count**. The only trace is a single `log.info` line — nothing
surfaces in the API response or the UI, so from the user's side it simply looks like "I enabled the
rule and nothing happens."
**Two combinations we actually hit**
| Case | replicas | step | min | max | computed | result |
| --- | --- | --- | --- | --- | --- | --- |
| Blocked by lower bound | 1 | 1 | 6 | 12 | 1 + 1 = 2 | 2 < 6 → cancelled |
| Blocked by upper bound | 1 | 6 | 1 | 5 | 1 + 6 = 7 | 7 > 5 → cancelled |
The upper-bound case has no way out at all: there is no value satisfying both `replicas ≤ 5` and
`replicas + 6 ≤ 5`, so scale-out is always cancelled regardless of the current replica count.
### To Reproduce
1. Create a deployment and leave `replicas = 1`.
2. Attach an autoscaling rule (metric source: `PROMETHEUS`).
- `step_size = 6`, `min_replicas = 1`, `max_replicas = 5`
- `scale_up_threshold = 0.9`, `scale_down_threshold = 0.5`, `cooldown = 30`
3. Push the metric above the threshold (e.g. 11 queued jobs → `github_actions_queued_jobs = 11`).
4. Wait for the rule evaluation interval.
**Observed** — `replicas` stays at 1 and `endpoint_auto_scaling_rules.last_triggered_at` is never
updated. Only the manager log shows `new count 7 above max 5`.
The same behavior reproduces on the lower-bound side with `step_size = 1`, `min_replicas = 6`.
### Expected Behavior
The computed value should be **clamped** to the boundary so that scaling proceeds as far as it can.
```python
new_replica_count = current + direction * step_size
if rule.action.max_replicas is not None:
new_replica_count = min(new_replica_count, rule.action.max_replicas)
if rule.action.min_replicas is not None:
new_replica_count = max(new_replica_count, rule.action.min_replicas)
if new_replica_count == current_replica_count:
continue # skip only when there is actually no change
```
With the example above, `1 + 6 = 7` would be clamped by `max 5` and scale **1 → 5**.
The meaning of `min_replicas` needs clarification as well. Today it acts only as a lower bound on the
*result of a scaling action*, so a deployment with `replicas = 1` and `min_replicas = 6` is never
raised to 6. The name reads as "minimum number of replicas to keep," which does not match that behavior.
### Anything else?
_No response_
Contributor guide
Research direction
Start in repositories/deployment/repository.py:1081-1101 and trace the autoscaling action calculation for min_replicas and max_replicas. Reproduce the lower- and upper-bound cases from the issue, then add regression coverage for clamping and for skipping only when the count is unchanged. Done means threshold-triggered scaling reaches the applicable boundary and updates as expected.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- prometheus, python
- Domain
- backend, cloud
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100