microsoft / microsoft/onnxruntime-genai

[.NET] Genny chat-bot sample doesn't support DirectML and Phi-3

Open
#569 10 comments 0 reactions 1 assignee Claimed by @PatriceVignola View on GitHub
ep:DML
Dominant language
C++
Stars
1.1k
Forks
354
Avg merge
2d 16h
Merged PRs (30d)
85

Description

I would like to update ..examples\csharp\Genny sample to support DirectML and Phi-3 model.

I managed to do it for Stateless mode (#568), however I faces with an issue for Stateful mode:

Inference fails with OnnxRuntimeGenAIException: 'Non-zero status code returned while running DmlFusedNode_0_0 node. Name:'DmlFusedNode_0_0' Status Message: invalid unordered_map key'

This happens due to current implementation of `private void AddPastTokens(Sequences sequences)` method in Genny sample:

```csharp
// Only keep (context_length - max_length) worth of history
while (_pastTokens.Count > ModelOptions.ContextLength - SearchOptions.MaxLength)
{
_pastTokens.RemoveAt(0);
}
```

for Phi3 both ModelOptions.ContextLength and SearchOptions.MaxLength are equal to 4096, so this method removes all tokens from session history and passes empty collection of tokens to `generatorParams.SetInputIDs`.

Should it be just `while (_pastTokens.Count > ModelOptions.ContextLength)` instead so number of passed tokens doesn't exceed model ContextLength?

Documention on GenAI C# API (https://onnxruntime.ai/docs/genai/api/csharp.html) doesn't provide any description on what each SearchOptions fields mean and how SetInputIDs works, so I don't have a clear view what this check is aimed to achive.

Could you please provide more info? If my understanding is correct, fix can be applied. If you approve it, I'll finish #568 PR

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.