v0.2.0(cuda12)对比 v0.1.13(cuda11)表现下降
- Dominant language
- Python
- Stars
- 1.3k
- Forks
- 275
- Avg merge
- 3d 17h
- Merged PRs (30d)
- 33
Description
环境配置
A 环境 cuda12.1 v0.2.0
B 环境 cuda11.8 v0.1.13
硬件
A800单卡测试
模型 qwen14B
单卡加载 int8推理 环境变量如下配置
export CUDA_VISIBLE_DEVICES=1
export MODEL_TYPE=qwen_2
export ACT_TYPE=BF16
export WEIGHT_TYPE=INT8
export INT8_KV_CACHE=1
export MAX_SEQ_LEN=32000
export CONCURRENCY_LIMIT=50
export TOKENIZER_PATH="/data/models/Qwen1.5-14B-Chat"
export CHECKPOINT_PATH="/data/models/Qwen1.5-14B-Chat"
export START_PORT=8020
export KV_CACHE_MEM_MB=8000
export PP_SIZE=1
export TP_SIZE=1
python -m maga_transformer.start_server
测试数据 10输入 50输出 超短场景
经测试 deepseek 等其余模型也有一定的速度下降
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the comparison using the A800, Qwen1.5-14B-Chat, environment variables, and `python -m maga_transformer.start_server` command given in the issue. Compare v0.2.0 with CUDA 12.1 against v0.1.13 with CUDA 11.8 using the reported 10-input, 50-output workload, then identify and document the source of the performance regression.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, machine-learning, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100