vllm-project / vllm-project/aibrix

Consider how cold start manager claim the pod from autoscaler or warm pool

Open
#8 0 comments 0 reactions 0 assignees View on GitHub
area/autoscaling
Dominant language
Go
Stars
5.1k
Forks
694
Avg merge
1d 19h
Merged PRs (30d)
98

Description

When the model scheduler likes to schedule the models, it needs to take the empty pod and place the model. However, empty pods normally comes from autoscaling, which is controlled in separate loop and cold start manager hard to know whenever it can get the pod.

We need a imperative way to claim the pod and wait for the pod ready which means we need more interactions with autoscaling logic, at least the resource claimer should know whether it can get the resource for that duration. This should not break autoscaler as well. The initial though is to introduce the warm pool concept we implement earlier but that's still not a perfect concept because it has not integrate with node level autoscaling yet.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the cold start manager, resource claimer, autoscaling logic, and earlier warm pool implementation described in the issue. Clarify how a pod is claimed and held while becoming ready, how node-level autoscaling participates, and what interactions must remain compatible with the autoscaler. Done means the ownership and readiness flow is defined and the autoscaler is not broken.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
cloud, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.