KPA scale-down killing a random pod
- Dominant language
- Go
- Stars
- 6.1k
- Forks
- 1.2k
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 2
Description
When pods are working hard, a scale down of a Service should not send kill signals to those pods, as it causes unnecessary overhead concerning setup/teardown and puts extra pressure on the system.
Proposal: set `PodDeletionCost` on pods currently handling requests:
Currently [available in beta](https://github.com/kubernetes/kubernetes/issues/107598) since k8s 1.22, could knative temporarily add the `controller.kubernetes.io/pod-deletion-cost` annotation to pods that are currently handling requests?
This would prevent long running requests (e.g. a complex deep learning ensemble) from being killed by a ReplicaSet scaledown.
_Originally posted by @ddelange in https://github.com/knative/serving/issues/3886#issuecomment-1158727239_
Contributor guide
Research direction
The issue names no files or tests. Start by locating the Service scale-down path and how pods currently handling requests are identified; review Kubernetes PodDeletionCost support and its beta availability. Done means request-serving pods receive the proposed annotation so ReplicaSet scale-down avoids terminating them during long-running requests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100