vllm-project / vllm-project/aibrix

StormService not Rollout unscheduled pods firstly

Open
#1,855 2 comments 0 reactions 1 assignee Claimed by @googs1025 View on GitHub
area/stormservice
Dominant language
Go
Stars
5.1k
Forks
697
Avg merge
1d 19h
Merged PRs (30d)
104

Description

Recently I use StormService in our production K8S, I found the Rollout pods rollback does not prioritize rolling back unscheduled pods, but rolls
back pods in sequence.

For example, I apply 6 replicas with nodeSelector app=aibrix, pod-0,3,4,5 is scheduled, 1 and 2 is not scheduled, because of only 4 GPU nodes with label app=aibrix are available.

Image

Then I unset label of 2 nodes for other usage temporarily. When I rolleout 6 replicas to 4 replicas, StormService Controller will roll back pod-5 and pod-4, the unscheduled pod-1 and pod-2 is not deleted.

Image

I expect the unscheduled pods can be delete firstly. I want to ask, delete the pods in order is how is the design, or delete the unscheduled pods firstly are taken into account in the future?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.