microsoft / microsoft/onnxruntime-genai

OnnxRuntimeGenAIChatClientOptions support DeepSeek-r1-Distill-14b

Open
#1,380 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
1.1k
Forks
354
Avg merge
2d 16h
Merged PRs (30d)
85

Description

I am not sure if the OnnxRuntimeGenAIChatClientOptions of the following deepseek model is correct. The value of message.Role.Value is fixed to "user". The user prompt tag of this deepseek model is < | User | >.
model:https://huggingface.co/onnxruntime/DeepSeek-R1-Distill-ONNX/tree/main/deepseek-r1-distill-qwen-14B/gpu/gpu-int4-rtn-block-32
```
public static OnnxRuntimeGenAIChatClientOptions GetPhi4() => new()
{
StopSequences = ["<|system|>", "<|user|>", "<|assistant|>", "<|end|>"],
PromptFormatter = static (messages, _) =>
{
var prompt = new StringBuilder();
foreach (var message in messages)
foreach (var content in message.Contents.OfType())
prompt.Append($"<|{message.Role.Value}|>\n{content.Text}<|end|>\n");
return prompt.Append("<|assistant|>\n").ToString();
}
};

public static OnnxRuntimeGenAIChatClientOptions GetDeepSeek() => new()
{
StopSequences = ["<|end▁of▁sentence|>"],
PromptFormatter = static (messages, _) =>
{
var prompt = new StringBuilder();
foreach (var message in messages)
foreach (var content in message.Contents.OfType())
prompt.Append($"<|begin▁of▁sentence|><|{message.Role.Value}|>\n{content.Text}<|end▁of▁sentence|>\n");
return prompt.Append("<|Assistant|>\n").ToString();
}
};
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the OnnxRuntimeGenAIChatClientOptions entry point and the GetDeepSeek formatter shown in the issue. Compare its role and prompt tokens with the linked DeepSeek-R1-Distill model format. Done means the options produce the model's expected user and assistant tags and stop sequence.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
machine-learning
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.