microsoft / microsoft/onnxruntime-genai

TheUnhandled exception. The parameter is incorrect.

Open
#883 0 comments 0 reactions 1 assignee Claimed by @PatriceVignola View on GitHub
ep:DML platform:windows
Dominant language
C++
Stars
1.1k
Forks
354
Avg merge
2d 16h
Merged PRs (30d)
85

Description

**Describe the bug**

Running `directml/directml-int4-awq-block-128` from `microsoft/Phi-3-mini-4k-instruct-onnx` using `LabsPhi301` base on Windows on Arm on Windows Dev Kit 2023 results in error:

```
dotnet run
unknown ARM CPU part 0xd4b ignored
unknown ARM CPU part 0xd4b ignored
unknown ARM CPU part 0xd4b ignored
unknown ARM CPU part 0xd4b ignored
unknown ARM CPU part 0xd4c ignored
unknown ARM CPU part 0xd4c ignored
unknown ARM CPU part 0xd4c ignored
unknown ARM CPU part 0xd4c ignored
Ask your question. Type an empty string to exit.

Q: Why is Arm advantageous over x86?
Phi3: TheUnhandled exception. Microsoft.ML.OnnxRuntimeGenAI.OnnxRuntimeGenAIException: D:\a\_work\1\onnxruntime-genai\src\dml\dml_update_mask_kernel.cpp(72)\onnxruntime-genai.DLL!00007FFDACF331E0: (caller: 00007FFDACF13880) Exception(1) tid(4dd0) 80070057 The parameter is incorrect.

at Microsoft.ML.OnnxRuntimeGenAI.Generator.ComputeLogits()
at Program.$(String[] args) in D:\Program.cs:line 63

```

**To Reproduce**
Steps to reproduce the behavior:
1. `git clone https://github.com/microsoft/Phi-3CookBook/`
2. Navigate to `/md/07.Labs/Csharp/src/LabsPhi301`
3. Set `generatorParams.SetSearchOption("past_present_share_buffer", true);` per #863
4. Copy `directml/directml-int4-awq-block-128` from `microsoft/Phi-3-mini-4k-instruct-onnx` into place
5. `dotnet restore`
6. `dotnet run`

**Expected behavior**

Be able to chat with `directml-int4-awq-block-128` build of Phi-3-mini-4k-instruct.

**Screenshots**

![image](https://github.com/user-attachments/assets/ad94be84-ae27-43af-a6f7-93861dfb9c5d)

**Desktop (please complete the following information):**
- Windows 11 on Arm
- Windows Dev Kit 2023
- .NET 8 latest SDK

**Additional context**

Program.cs:

```
using Microsoft.ML.OnnxRuntimeGenAI;

var modelPath = Path.Combine(AppContext.BaseDirectory, @"Phi-3-mini-4k-instruct-onnx\directml-int4-awq-block-128");
var model = new Model(modelPath);
var tokenizer = new Tokenizer(model);

var systemPrompt = "You are an AI assistant that helps people find information. Answer questions using a direct style. Do not share more information than is requested by the users.";

// chat start
Console.WriteLine(@"Ask your question. Type an empty string to exit.");

// chat loop
while (true)
{
// Get user question
Console.WriteLine();
Console.Write(@"Q: ");
var userQ = Console.ReadLine();
if (string.IsNullOrEmpty(userQ))
{
break;
}

// show phi3 response
Console.Write("Phi3: ");
var fullPrompt = $"<|system|>{systemPrompt}<|end|><|user|>{userQ}<|end|><|assistant|>";
var tokens = tokenizer.Encode(fullPrompt);

var generatorParams = new GeneratorParams(model);
generatorParams.SetSearchOption("max_length", 1048);
generatorParams.SetSearchOption("past_present_share_buffer", true);
generatorParams.SetInputSequences(tokens);

var generator = new Generator(model, generatorParams);
while (!generator.IsDone())
{
generator.ComputeLogits();
generator.GenerateNextToken();
var outputTokens = generator.GetSequence(0);
var newToken = outputTokens.Slice(outputTokens.Length - 1, 1);
var output = tokenizer.Decode(newToken);
Console.Write(output);
}
Console.WriteLine();
}
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.