microsoft / microsoft/WindowsAppSDK
Phi Silica text completions are really slow
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 4.7k
- Forks
- 471
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 28
Description
While embedding performance is OK (about 2,000 tokens/second), text completions are crazy slow: **5 seconds for roughly 250 prompt tokens and 20 completion tokens.**
I don't think my use case is too crazy -- it's a subset of a much larger AI problem (measured in tokens & desired intelligence) we are effectively solving with GPT 4o mini today.
The answer is borderline workable and will need lots of prompt engineering. But even if it nailed it, for this to be workable in a UI and to choose Phi Silica over cloud models for this specific use case, I need it under 1 second -- ideally under 500ms.
Are these chat completion token counts in the ballpark for what this feature is designed? Am I missing some key configuration, a model selector, or something? What can I expect out of Phi Silica?
For reference, my code looks roughly like:
```csharp
using Microsoft.Windows.AI.Generative;
if (!LanguageModel.IsAvailable()) await LanguageModel.MakeAvailableAsync();
using LanguageModel languageModel = await LanguageModel.CreateAsync();
var prompt = "...";
var options = new LanguageModelOptions();
options.Temp = ...;
options.Top_p = ...;
// I run the next line over 10 to 100 iterations, and average the iteration time:
var result = await languageModel.GenerateResponseAsync(options, prompt);
```
My setup is:
- Dell Latitude 7455 with Snapdragon X Elite (X1E80100)
- Windows 26120.3863
- Microsoft.WindowsAppSDK v1.7.250127003-experimental3
- .NET 8.0
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the GenerateResponseAsync call in the reported C# sample and reproduce the timing on the listed Snapdragon X Elite and Windows versions. Compare the roughly 250-token prompt and 20-token completion across the 10–100 iteration benchmark, then establish whether configuration or model selection changes the result. Done means documenting expected performance and any confirmed configuration or platform issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp
- Domain
- ai, desktop
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100