vllm-project / vllm-project/aibrix
StreamLoader splits tensors more evenly
Open
area/acceleration
area/model-loader
kind/enhancement
- Dominant language
- Go
- Stars
- 5.1k
- Forks
- 694
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 98
Description
### 🚀 Feature Description and Motivation
The current way of loading tensors in StreamLoader is tensor by tensor, and the total size distribution of tensors processed in each thread is not evenly distributed, which can cause the largest thread to become a bottleneck in speed. If the tensor size pulled by each thread is similar, it can ensure an increase in bandwidth usage.
### Use Case
_No response_
### Proposed Solution
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.