MegEngine / MegEngine/InferLLM

在运行llama2-13b的时候出现以下问题

Open
#77 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
752
Forks
94
PR merge metrics
No merged PRs in 30d

Description

在微调过的llama2模型上能够完成转换和量化,但在运行模型的时候会报错
```
# ./llama -m ../llama2-13b-sft-filterd-v17/llama2-13b-sft-filterd-v17-inferllm-fp32.bin -g GPU --version 2
main: seed = 1709878763
total vocab length = 68419
weight tok_embeddings.weight is not match.
Assert ' weight->length() == nr_number ' failed at file : /InferLLM/src/core/graph.cpp
line 325 : virtual void inferllm::Graph::load(std::shared_ptr, inferllm::LlmParams&, std::shared_ptr),
extra message: Error length of weight is mismatch.Aborted (core dumped)
root@goedge_master:/InferLLM/build# ./llama -m llama2-13b-sft-filterd-v17-q4.bin -g GPU --version 2
main: seed = 1709878793
total vocab length = 68419
weight tok_embeddings.weight is not match.
Assert ' weight->length() == nr_number ' failed at file : /InferLLM/src/core/graph.cpp
line 325 : virtual void inferllm::Graph::load(std::shared_ptr, inferllm::LlmParams&, std::shared_ptr),
extra message: Error length of weight is mismatch.Aborted (core dumped)
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the failure using the llama commands and the fine-tuned Llama 2 model files named in the report. Start at InferLLM/src/core/graph.cpp line 325, then trace how tok_embeddings.weight is loaded and how the vocabulary length is determined. Done means both the FP32 and quantized models load without the length-mismatch assertion.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.