vllm-project / vllm-project/production-stack
Question about `tolerations` in `servingEngineSpec`
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2.6k
- Forks
- 503
- Avg merge
- 4d 17h
- Merged PRs (30d)
- 8
Description
In many cases, tolerations property is used together with nodeSelectorTerms.
However it seems tolerations is in servingEngineSpec while nodeSelectorTerms is in modelSpec.
I think it would be great the two properties are in the same section one of either servingEngineSpec or modelSpec.
Since those properties are applied per node, it would be better to have it in modelSpec I guess?
The downside might be the modelSpec could be too verbose.
On the other hand, it could cover the case where each model could be deployed to different tainted Nodes with different labels.
Could you consider this case?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the definitions and usage of servingEngineSpec, modelSpec, tolerations, and nodeSelectorTerms. Review how these properties are applied to Kubernetes workloads, then determine whether moving tolerations enables per-model node placement without making modelSpec excessively verbose; done means an agreed configuration design and corresponding behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes
- Domain
- infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100