MegEngine / MegEngine/InferLLM
在运行llama2-13b的时候出现以下问题
- Dominant language
- C++
- Stars
- 752
- Forks
- 94
- PR merge metrics
- No merged PRs in 30d
Description
在微调过的llama2模型上能够完成转换和量化,但在运行模型的时候会报错
```
# ./llama -m ../llama2-13b-sft-filterd-v17/llama2-13b-sft-filterd-v17-inferllm-fp32.bin -g GPU --version 2
main: seed = 1709878763
total vocab length = 68419
weight tok_embeddings.weight is not match.
Assert ' weight->length() == nr_number ' failed at file : /InferLLM/src/core/graph.cpp
line 325 : virtual void inferllm::Graph::load(std::shared_ptr, inferllm::LlmParams&, std::shared_ptr),
extra message: Error length of weight is mismatch.Aborted (core dumped)
root@goedge_master:/InferLLM/build# ./llama -m llama2-13b-sft-filterd-v17-q4.bin -g GPU --version 2
main: seed = 1709878793
total vocab length = 68419
weight tok_embeddings.weight is not match.
Assert ' weight->length() == nr_number ' failed at file : /InferLLM/src/core/graph.cpp
line 325 : virtual void inferllm::Graph::load(std::shared_ptr, inferllm::LlmParams&, std::shared_ptr),
extra message: Error length of weight is mismatch.Aborted (core dumped)
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the failure using the llama commands and the fine-tuned Llama 2 model files named in the report. Start at InferLLM/src/core/graph.cpp line 325, then trace how tok_embeddings.weight is loaded and how the vocabulary length is determined. Done means both the FP32 and quantized models load without the length-mismatch assertion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100