vllm-project / vllm-project/aibrix
StreamLoader supports each TP process to only read the corresponding part of the model file
Open
area/acceleration
area/model-loader
kind/enhancement
priority/critical-urgent
- Dominant language
- Go
- Stars
- 5.1k
- Forks
- 694
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 98
Description
### 🚀 Feature Description and Motivation
Currently, each TP (Tensor Parallism) process in StreamLoader reads all model files, resulting in duplicate file transfers and reads, which reduces the overall loading speed. If StreamLoader supports each TP process to only read the corresponding part of the model file, it will reduce the additional overhead of this part.
### Use Case
_No response_
### Proposed Solution
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.