推理速度很慢
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 12.3k
- Forks
- 1.1k
- PR merge metrics
- No merged PRs in 30d
Description
在32G V100显卡上进行了FP16精度模型多卡部署以及8 Bit和4 Bit量化模型单卡部署，发现推理速度都很慢。一个普通问题需要100-120秒甚至更长时间才能做出回答。
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue names no files, tests, or entry points. Begin by reproducing the reported FP16 multi-GPU and 8-bit/4-bit single-GPU deployments on a 32G V100, then profile a normal query to identify where the 100–120 second latency occurs. No completion target is specified beyond isolating the cause of the slow inference.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100