microsoft / microsoft/WindowsAppSDK

Phi Silica text completions are really slow

Open
#5,334 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area-WCR
Dominant language
C++
Stars
4.7k
Forks
471
Avg merge
3d 13h
Merged PRs (30d)
28

Description

While embedding performance is OK (about 2,000 tokens/second), text completions are crazy slow: **5 seconds for roughly 250 prompt tokens and 20 completion tokens.**

I don't think my use case is too crazy -- it's a subset of a much larger AI problem (measured in tokens & desired intelligence) we are effectively solving with GPT 4o mini today.

The answer is borderline workable and will need lots of prompt engineering. But even if it nailed it, for this to be workable in a UI and to choose Phi Silica over cloud models for this specific use case, I need it under 1 second -- ideally under 500ms.

Are these chat completion token counts in the ballpark for what this feature is designed? Am I missing some key configuration, a model selector, or something? What can I expect out of Phi Silica?

For reference, my code looks roughly like:

```csharp
using Microsoft.Windows.AI.Generative;

if (!LanguageModel.IsAvailable()) await LanguageModel.MakeAvailableAsync();

using LanguageModel languageModel = await LanguageModel.CreateAsync();

var prompt = "...";

var options = new LanguageModelOptions();
options.Temp = ...;
options.Top_p = ...;

// I run the next line over 10 to 100 iterations, and average the iteration time:
var result = await languageModel.GenerateResponseAsync(options, prompt);
```

My setup is:

- Dell Latitude 7455 with Snapdragon X Elite (X1E80100)
- Windows 26120.3863
- Microsoft.WindowsAppSDK v1.7.250127003-experimental3
- .NET 8.0

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the GenerateResponseAsync call in the reported C# sample and reproduce the timing on the listed Snapdragon X Elite and Windows versions. Compare the roughly 250-token prompt and 20-token completion across the 10–100 iteration benchmark, then establish whether configuration or model selection changes the result. Done means documenting expected performance and any confirmed configuration or platform issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
ai, desktop
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.