lablup / lablup/backend.ai

Autoscaling is silently skipped when step_size crosses min/max_replicas

Open
#13,848 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
17h 7m
Merged PRs (30d)
358

Description

### What Operating System(s) are you seeing this problem on?

Linux (x86-64)

### Backend.AI version

26.4.10

### Describe the bug

Even when an autoscaling rule's threshold condition is met, if `current_replicas ± step_size` falls
outside the `min_replicas` / `max_replicas` range, the value is **not clamped to the boundary — the
scaling action is cancelled entirely**.

`repositories/deployment/repository.py:1081-1101` treats the bounds as a skip condition rather than a clamp.

```python
new_replica_count = max(0, current_replica_count + scale_direction * rule.action.step_size)

if rule.action.min_replicas is not None and new_replica_count < rule.action.min_replicas:
log.info("AUTOSCALE(...): new count {} below min {}", ...)
continue # ← gives up on scaling

if rule.action.max_replicas is not None and new_replica_count > rule.action.max_replicas:
log.info("AUTOSCALE(...): new count {} above max {}", ...)
continue # ← gives up on scaling
```
As a result, any configuration where `step_size` straddles a boundary creates a lock-up in which
**scaling never happens from any replica count**. The only trace is a single `log.info` line — nothing
surfaces in the API response or the UI, so from the user's side it simply looks like "I enabled the
rule and nothing happens."

**Two combinations we actually hit**

| Case | replicas | step | min | max | computed | result |
| --- | --- | --- | --- | --- | --- | --- |
| Blocked by lower bound | 1 | 1 | 6 | 12 | 1 + 1 = 2 | 2 < 6 → cancelled |
| Blocked by upper bound | 1 | 6 | 1 | 5 | 1 + 6 = 7 | 7 > 5 → cancelled |

The upper-bound case has no way out at all: there is no value satisfying both `replicas ≤ 5` and
`replicas + 6 ≤ 5`, so scale-out is always cancelled regardless of the current replica count.

### To Reproduce

1. Create a deployment and leave `replicas = 1`.
2. Attach an autoscaling rule (metric source: `PROMETHEUS`).
- `step_size = 6`, `min_replicas = 1`, `max_replicas = 5`
- `scale_up_threshold = 0.9`, `scale_down_threshold = 0.5`, `cooldown = 30`
3. Push the metric above the threshold (e.g. 11 queued jobs → `github_actions_queued_jobs = 11`).
4. Wait for the rule evaluation interval.

**Observed** — `replicas` stays at 1 and `endpoint_auto_scaling_rules.last_triggered_at` is never
updated. Only the manager log shows `new count 7 above max 5`.

The same behavior reproduces on the lower-bound side with `step_size = 1`, `min_replicas = 6`.

### Expected Behavior

The computed value should be **clamped** to the boundary so that scaling proceeds as far as it can.

```python
new_replica_count = current + direction * step_size
if rule.action.max_replicas is not None:
new_replica_count = min(new_replica_count, rule.action.max_replicas)
if rule.action.min_replicas is not None:
new_replica_count = max(new_replica_count, rule.action.min_replicas)
if new_replica_count == current_replica_count:
continue # skip only when there is actually no change
```

With the example above, `1 + 6 = 7` would be clamped by `max 5` and scale **1 → 5**.

The meaning of `min_replicas` needs clarification as well. Today it acts only as a lower bound on the
*result of a scaling action*, so a deployment with `replicas = 1` and `min_replicas = 6` is never
raised to 6. The name reads as "minimum number of replicas to keep," which does not match that behavior.

### Anything else?

_No response_

Contributor guide

Open the contributing guide

Research direction

Start in repositories/deployment/repository.py:1081-1101 and trace the autoscaling action calculation for min_replicas and max_replicas. Reproduce the lower- and upper-bound cases from the issue, then add regression coverage for clamping and for skipping only when the count is unchanged. Done means threshold-triggered scaling reaches the applicable boundary and updates as expected.

Written by the indexing model from the issue text.

Assessment

Tech stack
prometheus, python
Domain
backend, cloud
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.