vllm-project / vllm-project/aibrix
Optimize StreamLoader process and thread count settings
Open
area/model-loader
kind/enhancement
priority/important-soon
- Dominant language
- Go
- Stars
- 5.1k
- Forks
- 694
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 98
Description
### 🚀 Feature Description and Motivation
We need some theoretical support between network IO and the number of processes and threads to optimize the settings of StreamLoader processes and threads. This can help us **automate** the determination of the appropriate number of processes and threads.
### Use Case
_No response_
### Proposed Solution
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.