microsoft / microsoft/onnxruntime-genai
Can't infer Qwen2-1.5B with a lora
- Dominant language
- C++
- Stars
- 1.1k
- Forks
- 354
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 85
Description
I use the Qwen2-1.5B model and a lora:
+ Convert Qwen2-1.5B
```bash
python3 builder.py -m Qwen2-1.5B-Instruct -o Qwen2-1.5B-Instruct-onnx-int4 -p int4 -e cpu --extra_options int4_block_size=128 int4_accuracy_level=4 int4_op_types_to_quantize=MatMul/Gather
```
+ Convert the lora
```bash
python -m olive convert-adapters -a ./release --adapter_format onnx_adapter -o ./release --log_level 4
```
+Infer code
```cpp
auto model = OgaModel::Create(path_model_dir.c_str());
auto lora = OgaAdapters::Create(*model);
lora->LoadAdapter("release.onnx_adapter", "best_lora");
auto tokenizer = OgaTokenizer::Create(*model);
auto tokenizer_stream = OgaTokenizerStream::Create(*tokenizer);
auto params = OgaGeneratorParams::Create(*model);
params->SetSearchOption("max_length", 128);
auto seq = OgaSequences::Create();
tokenizer->Encode(query.c_str(), *seq);
params->SetInputSequences(*seq);
auto generator = OgaGenerator::Create(*model, *params);
generator->SetActiveAdapter(*lora, "best_lora");
std::stringstream result_ss;
while (!generator->IsDone())
{
generator->ComputeLogits();
generator->GenerateNextToken();
const auto num_tokens = generator->GetSequenceCount(0);
const auto new_token = generator->GetSequenceData(0)[num_tokens - 1];
result_ss << tokenizer_stream->Decode(new_token);
}
```
but it dump !!! I don't know why
```bash
terminate called after throwing an instance of 'std::runtime_error'
what(): Invalid input name: model.layers.9.self_attn.v_proj.lora_B.weight
```
+ Version
1. onnxruntime-genai : 0.5.2
2. olive-ai 0.7.1.1
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the reported flow with builder.py, Olive's convert-adapters command, and the C++ inference sequence using onnxruntime-genai 0.5.2 and olive-ai 0.7.1.1. Investigate why loading best_lora produces the invalid model.layers.9.self_attn.v_proj.lora_B.weight input name, then verify that the adapter loads and inference completes without the runtime error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- ai, backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100