ByteDance-Seed / ByteDance-Seed/decoupleQ

关于量化模型推理

Open
#1 16 comments 0 reactions 0 assignees View on GitHub
Dominant language
Cuda
Stars
153
Forks
10
PR merge metrics
No merged PRs in 30d

Description

非常赞的工作!有两个问题问题请教一下:
1)使用 decoupleQ 量化后的LLaMA模型,推理性能如何?是否有对应一些测试数据?
2)量化后的模型如何部署进行推理呢? README 中,有看到 NVIDIA/TensorRT-LLM#1568,是直接使用 TensorRT-LLM 进行部署吗?然后请问有对应的推理部署脚本吗?

谢谢!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the README references to decoupleQ, LLaMA, and NVIDIA/TensorRT-LLM#1568. Determine whether the repository contains benchmark results or deployment scripts, then document the available inference performance and deployment path; if those artifacts are absent, the scope needs clarification before work begins.

Written by the indexing model from the issue text.

Assessment

Domain
ai, devops, machine-learning
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.