knative / knative/serving

Properly Handle Pods with schedulingGates

Open
#15,308 9 comments 0 reactions 0 assignees View on GitHub
area/autoscale kind/feature lifecycle/frozen
Dominant language
Go
Stars
6.1k
Forks
1.2k
Avg merge
2d 7h
Merged PRs (30d)
2

Description

/area autoscale

## Describe the feature
I would like to request to appropriately handle the Pods with [shcedulingGates](https://kubernetes.io/docs/concepts/scheduling-eviction/pod-scheduling-readiness/) (a.k.a gated pods).
I'm wondering if the KPA should not scale deployment with gated pods into zero.

Now, as a workaround, we can avoid the accident automatically scale-in by setting the maximum value (2147483647) in the `progressDeadlineSeconds`.

## background

Currently, the Knative Pod Autoscaler (KPA) automatically scales the not ready pods due to unschedulable into zero based on the `progressDeadlineSeconds` to improve better resource consumption, and the KPA doesn't consider the reason why the Pods are not schedulable.
Additionally, we can not specify null in the `progressDeadlineSeconds` because the field is propagated to Deployment as well.

https://github.com/knative/serving/blob/3aa9210a0d1368c01ec6a7ccfa9e3db5a564a5a5/pkg/reconciler/autoscaling/kpa/scaler.go#L199-L204

https://github.com/knative/serving/blob/3aa9210a0d1368c01ec6a7ccfa9e3db5a564a5a5/pkg/deployment/config.go#L102

However, the unschedulable phase due to the scheduling gate is intentional behavior because the external controller, like the quota management/queuing system, stops scheduling the Pods into any Nodes.

Based on the current KPA implementation, once the gated pod is removed by scaled deployment into zero, the quota management/queueing system could never pop the Pods into the cluster.

Contributor guide

Open the contributing guide

Research direction

Start by reading the KPA scaling logic in pkg/reconciler/autoscaling/kpa/scaler.go around the linked lines, then review progressDeadlineSeconds handling in pkg/reconciler/autoscaling/config.go. Trace how gated Pods are treated when deployments scale toward zero, and define tests that demonstrate the intended behavior for Pods with schedulingGates without losing the existing unschedulable-pod handling.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
backend, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.