KPA cannot scale gang workloads to zero
- Dominant language
- Go
- Stars
- 6.1k
- Forks
- 1.2k
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 2
Description
/area autoscale
/kind bug
## What version of Knative?
v1.23.0
## Expected Behavior
KPA should support gang workloads whose valid replica states are:
```text
replicas == 0, or replicas >= minCount
```
## Actual Behavior
KPA gradually scales `2 -> 1 -> 0`. With the default
`MaxScaleDownRate=2`:
```text
Ready=2, metricDesired=0
desired=floor(2/2)=1
```
A gang workload with `minCount=2` cannot realize one replica, so it remains at
two. KPA continues observing two Ready Pods and repeatedly requests one,
preventing scale-to-zero:
```text
KPA requests 1 -> target remains at 2 -> repeat
```
Pods remain stable, but KPA desired remains `1`, actual remains `2`, and a
persistent write loop occurs. A control allowing one replica successfully
converges through `2 -> 1 -> 0`.
Related behavior is discussed in #14017 and #15154.
KEDA avoids this using separate idle and active floors:
```yaml
idleReplicaCount: 0
minReplicaCount: 2
```
https://keda.sh/docs/2.20/reference/scaledobject-spec/#idlereplicacount
This matters for gang-aware workload controllers, for example
ai-dynamo/grove#686, and Kubernetes Workload/PodGroup APIs in KEP-4671:
https://github.com/ai-dynamo/grove/pull/686
https://github.com/kubernetes/enhancements/tree/master/keps/sig-scheduling/4671-gang-scheduling
Could KPA support a similar separation between idle and active floors?
Contributor guide
Research direction
Start by tracing the KPA scale-down decision that computes desired replicas from Ready pods and review related issues #14017 and #15154. Compare the requested idle and active floor separation with KEDA's documented behavior, then define convergence tests showing that a gang workload can move from its active minimum to zero without a persistent write loop.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- backend, infrastructure
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100