knative / knative/serving

KPA scale-down killing a random pod

Open
#13,075 25 comments 1 reaction 0 assignees View on GitHub
lifecycle/frozen triage/accepted
Dominant language
Go
Stars
6.1k
Forks
1.2k
Avg merge
2d 7h
Merged PRs (30d)
2

Description

When pods are working hard, a scale down of a Service should not send kill signals to those pods, as it causes unnecessary overhead concerning setup/teardown and puts extra pressure on the system.

Proposal: set `PodDeletionCost` on pods currently handling requests:

Currently [available in beta](https://github.com/kubernetes/kubernetes/issues/107598) since k8s 1.22, could knative temporarily add the `controller.kubernetes.io/pod-deletion-cost` annotation to pods that are currently handling requests?

This would prevent long running requests (e.g. a complex deep learning ensemble) from being killed by a ReplicaSet scaledown.

_Originally posted by @ddelange in https://github.com/knative/serving/issues/3886#issuecomment-1158727239_

Contributor guide

Open the contributing guide

Research direction

The issue names no files or tests. Start by locating the Service scale-down path and how pods currently handling requests are identified; review Kubernetes PodDeletionCost support and its beta availability. Done means request-serving pods receive the proposed annotation so ReplicaSet scale-down avoids terminating them during long-running requests.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
backend
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.