microsoft / microsoft/onnxruntime-genai

Can't infer Qwen2-1.5B with a lora

Open
#1,186 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
1.1k
Forks
354
Avg merge
2d 16h
Merged PRs (30d)
85

Description

I use the Qwen2-1.5B model and a lora:

+ Convert Qwen2-1.5B
```bash
python3 builder.py -m Qwen2-1.5B-Instruct -o Qwen2-1.5B-Instruct-onnx-int4 -p int4 -e cpu --extra_options int4_block_size=128 int4_accuracy_level=4 int4_op_types_to_quantize=MatMul/Gather
```

+ Convert the lora
```bash
python -m olive convert-adapters -a ./release --adapter_format onnx_adapter -o ./release --log_level 4
```

+Infer code
```cpp
auto model = OgaModel::Create(path_model_dir.c_str());
auto lora = OgaAdapters::Create(*model);
lora->LoadAdapter("release.onnx_adapter", "best_lora");
auto tokenizer = OgaTokenizer::Create(*model);
auto tokenizer_stream = OgaTokenizerStream::Create(*tokenizer);
auto params = OgaGeneratorParams::Create(*model);
params->SetSearchOption("max_length", 128);
auto seq = OgaSequences::Create();
tokenizer->Encode(query.c_str(), *seq);
params->SetInputSequences(*seq);
auto generator = OgaGenerator::Create(*model, *params);
generator->SetActiveAdapter(*lora, "best_lora");
std::stringstream result_ss;
while (!generator->IsDone())
{
generator->ComputeLogits();
generator->GenerateNextToken();
const auto num_tokens = generator->GetSequenceCount(0);
const auto new_token = generator->GetSequenceData(0)[num_tokens - 1];
result_ss << tokenizer_stream->Decode(new_token);
}
```
but it dump !!! I don't know why
```bash
terminate called after throwing an instance of 'std::runtime_error'
what(): Invalid input name: model.layers.9.self_attn.v_proj.lora_B.weight
```
+ Version
1. onnxruntime-genai : 0.5.2
2. olive-ai 0.7.1.1

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the reported flow with builder.py, Olive's convert-adapters command, and the C++ inference sequence using onnxruntime-genai 0.5.2 and olive-ai 0.7.1.1. Investigate why loading best_lora produces the invalid model.layers.9.self_attn.v_proj.lora_B.weight input name, then verify that the adapter loads and inference completes without the runtime error.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
ai, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.