knative / knative/serving

KPA cannot scale gang workloads to zero

Open
#16,684 1 comment 0 reactions 0 assignees View on GitHub
area/autoscale kind/bug
Dominant language
Go
Stars
6.1k
Forks
1.2k
Avg merge
2d 7h
Merged PRs (30d)
2

Description

/area autoscale
/kind bug

## What version of Knative?

v1.23.0

## Expected Behavior

KPA should support gang workloads whose valid replica states are:

```text
replicas == 0, or replicas >= minCount
```

## Actual Behavior

KPA gradually scales `2 -> 1 -> 0`. With the default
`MaxScaleDownRate=2`:

```text
Ready=2, metricDesired=0
desired=floor(2/2)=1
```

A gang workload with `minCount=2` cannot realize one replica, so it remains at
two. KPA continues observing two Ready Pods and repeatedly requests one,
preventing scale-to-zero:

```text
KPA requests 1 -> target remains at 2 -> repeat
```

Pods remain stable, but KPA desired remains `1`, actual remains `2`, and a
persistent write loop occurs. A control allowing one replica successfully
converges through `2 -> 1 -> 0`.

Related behavior is discussed in #14017 and #15154.

KEDA avoids this using separate idle and active floors:

```yaml
idleReplicaCount: 0
minReplicaCount: 2
```

https://keda.sh/docs/2.20/reference/scaledobject-spec/#idlereplicacount

This matters for gang-aware workload controllers, for example
ai-dynamo/grove#686, and Kubernetes Workload/PodGroup APIs in KEP-4671:

https://github.com/ai-dynamo/grove/pull/686
https://github.com/kubernetes/enhancements/tree/master/keps/sig-scheduling/4671-gang-scheduling

Could KPA support a similar separation between idle and active floors?

Contributor guide

Open the contributing guide

Research direction

Start by tracing the KPA scale-down decision that computes desired replicas from Ready pods and review related issues #14017 and #15154. Compare the requested idle and active floor separation with KEDA's documented behavior, then define convergence tests showing that a gang workload can move from its active minimum to zero without a persistent write loop.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
backend, infrastructure
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.