ByteDance-Seed / ByteDance-Seed/decoupleQ
关于量化模型推理
Open
- Dominant language
- Cuda
- Stars
- 153
- Forks
- 10
- PR merge metrics
- No merged PRs in 30d
Description
非常赞的工作!有两个问题问题请教一下:
1)使用 decoupleQ 量化后的LLaMA模型,推理性能如何?是否有对应一些测试数据?
2)量化后的模型如何部署进行推理呢? README 中,有看到 NVIDIA/TensorRT-LLM#1568,是直接使用 TensorRT-LLM 进行部署吗?然后请问有对应的推理部署脚本吗?
谢谢!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the README references to decoupleQ, LLaMA, and NVIDIA/TensorRT-LLM#1568. Determine whether the repository contains benchmark results or deployment scripts, then document the available inference performance and deployment path; if those artifacts are absent, the scope needs clarification before work begins.
Written by the indexing model from the issue text.
Assessment
- Domain
- ai, devops, machine-learning
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100