vllm-project / vllm-project/aibrix
Consider how cold start manager claim the pod from autoscaler or warm pool
- Dominant language
- Go
- Stars
- 5.1k
- Forks
- 694
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 98
Description
When the model scheduler likes to schedule the models, it needs to take the empty pod and place the model. However, empty pods normally comes from autoscaling, which is controlled in separate loop and cold start manager hard to know whenever it can get the pod.
We need a imperative way to claim the pod and wait for the pod ready which means we need more interactions with autoscaling logic, at least the resource claimer should know whether it can get the resource for that duration. This should not break autoscaler as well. The initial though is to introduce the warm pool concept we implement earlier but that's still not a perfect concept because it has not integrate with node level autoscaling yet.
Contributor guide
Research direction
Start by tracing the cold start manager, resource claimer, autoscaling logic, and earlier warm pool implementation described in the issue. Clarify how a pod is claimed and held while becoming ready, how node-level autoscaling participates, and what interactions must remain compatible with the autoscaler. Done means the ownership and readiness flow is defined and the autoscaler is not broken.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- cloud, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100