microsoft / microsoft/foundry-local

[Feature Request] Add Kokoro-82M text-to-speech support

Open
#958 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
2.6k
Forks
369
Avg merge
2d 17h
Merged PRs (30d)
39

Description

## Describe the feature request

Add text-to-speech (TTS) as a supported Foundry Local task and onboard **Kokoro-82M v1.0** into the curated model catalog.

Foundry Local currently exposes speech-to-text models such as Whisper, Nemotron ASR, and Parakeet, but no model that produces speech. Kokoro is a particularly good fit for on-device inference:

- 82 million parameters
- Apache-2.0-licensed model weights
- Existing ONNX Runtime conversions with fp32, fp16, and quantized variants
- Approximately 86-326 MB depending on quantization
- Existing CPU and WebGPU deployments
- Multiple voices and languages

Relevant sources:

- Official model card: https://huggingface.co/hexgrad/Kokoro-82M
- ONNX v1.0 conversion: https://huggingface.co/onnx-community/Kokoro-82M-v1.0-ONNX
- NVIDIA-optimized ONNX model: https://huggingface.co/nvidia/kokoro-82M-onnx-opt
- ONNX Runtime wrapper: https://github.com/thewh1teagle/kokoro-onnx

The catalog artifact would need to include or acquire the required tokenizer/phonemizer assets and voice embeddings in addition to the ONNX graph. The team should validate or reproduce the conversion rather than necessarily consuming a community artifact directly.

This also requires a supported TTS inference surface. A possible shape, without prescribing the design, could include:

- A `text-to-speech` task type
- An SDK speech-generation client/session
- An OpenAI-compatible `/v1/audio/speech` endpoint
- CLI support for generating a WAV file from text
- CPU and WebGPU variants across supported desktop platforms

## Describe scenario use case

Applications should be able to synthesize speech fully on-device using the same runtime and model-management experience already used for chat, embeddings, multimodal inference, and transcription.

Example scenarios include:

- Accessibility and read-aloud experiences
- Private/offline voice assistants
- Spoken notifications and summaries
- Low-latency conversational applications combining local STT, an LLM, and TTS
- Cross-platform desktop applications that should not require a second inference runtime

Today these applications must integrate a separate Kokoro runtime or use a cloud TTS service, losing Foundry Local's unified hardware selection, caching, packaging, and offline experience.

## Customer requirement and impact

TTS completes the local conversational loop: **speech input -> local reasoning -> speech output**. Kokoro offers a small, permissively licensed starting model that is practical for consumer devices and substantially lighter than generative TTS models in the billion-parameter range.

Adding it would provide immediate value to accessibility, assistant, productivity, and privacy-sensitive applications while establishing the runtime/API foundation for future speech-synthesis models.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Review the existing speech-to-text task and model-management experience first, then trace the proposed /v1/audio/speech endpoint and CLI generation flow. Done means defining and implementing a supported TTS surface, onboarding Kokoro-82M with its tokenizer, phonemizer, and voice assets, and validating CPU and WebGPU variants across supported desktop platforms.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
api, cli, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.