microsoft / microsoft/foundry-local

[Bug] Whisper Base Small model with accuracy issue and service not stable issue

Open
#1,037 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
C++
Stars
2.6k
Forks
369
Avg merge
2d 17h
Merged PRs (30d)
39

Description

### Describe the issue

One ISV team is developing local AI features based on FL Whisper model, they want to know whether we can help them on the fixing and improvement.
Model : Whisper Base /Small
1. We often observe word segmentation issues in the transcription results, where unnecessary spaces are inserted. For example, "2006年" may be transcribed as "200 6年". Do you have any recommended optimization approaches for improving tokenization and text formatting accuracy?
2. During testing, we occasionally encounter a situation where the model service becomes unresponsive (typically after repeatedly starting and stopping the feature). Once this occurs, the service can no longer be accessed successfully until the PC is rebooted.
o Is there an API available to forcibly restart the LLM service?
o Is there a recommended minimum interval between service startup and shutdown operations to avoid this issue?

### To reproduce

1.We often observe word segmentation issues in the transcription results, where unnecessary spaces are inserted. For example, "2006年" may be transcribed as "200 6年". Do you have any recommended optimization approaches for improving tokenization and text formatting accuracy?
2. During testing, we occasionally encounter a situation where the model service becomes unresponsive (typically after repeatedly starting and stopping the feature). Once this occurs, the service can no longer be accessed successfully until the PC is rebooted.

### Urgency

_No response_

### Platform and architecture

Windows X64

### OS Version

Windows 11 25H2 x64 - 26200.9168

### Installation type

Released package/binary

### Foundry Local version

SDK1.2.3

### API or surface area

C# SDK

### Hardware acceleration/backend

CPU

### Backend/runtime version

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with the C# SDK's Whisper Base/Small service lifecycle and transcription API on Windows x64; reproduce the spacing error and repeated start/stop failure using SDK 1.2.3 on CPU. Done means a defined mitigation for the "2006年" segmentation and a reliable restart or lifecycle recommendation that avoids requiring a reboot.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
ai, api
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.