OpenBMB / OpenBMB/VoxCPM

Is VoxCPM2 planned for MiniCPM-o? — and if not, the path to real-time Korean speech

Open
#307 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
37.8k
Forks
4.3k
Avg merge
7m
Merged PRs (30d)
1

Description

We've been building a Korean real-time voice assistant on MiniCPM-o 4.5. We chose MiniCPM-o specifically for its live full-duplex behavior: it produces speech token-interleaved as it generates, so it can start talking immediately and be interrupted mid-sentence. That real-time feel is the core of what we're building — not a nice-to-have.

MiniCPM-o's Korean speech wasn't there out of the box, so we fine-tuned its native Talker for Korean. This has been real, ongoing work and it was paying off — our Korean Talker CER went from ~0.33 to ~0.14 and was still improving.

Then VoxCPM2 came out. Its Korean quality with reference-voice cloning is excellent, and since it's also yours, we tried it as MiniCPM-o's voice. As standalone TTS it's great — but it's batch/utterance synthesis. Inside MiniCPM-o's token-interleaved loop it either comes out disjoint (each small fragment synthesized with its own terminal prosody), or, if we buffer whole clauses first, it adds several seconds of silence before speech and barge-in is gone. (We've seen the earlier streaming issues, e.g. #146 / #214 — our question is specifically about VoxCPM2 as the voice of a token-interleaved full-duplex LLM, not standalone streaming quality.) So VoxCPM2 as the live voice loses exactly the real-time property we built everything around.

That leaves us at a real fork, and only you can answer the first part:

  1. Is there a plan to bring VoxCPM2 into MiniCPM-o (as its voice, or a streaming integration)? If something official is coming, we'd rather wait for it than keep pushing our own Talker training in parallel.
  2. If there's no such plan — is there a way to get real-time speech within MiniCPM-o itself that we're missing (a streaming path for VoxCPM2-level quality, or improving the native vocoder)?
  3. If neither — is the path we're on the right one in your view: continuing to fine-tune the native Korean Talker, plus improving the Thinker for Korean?

We've done the real work here and can share detail and metrics — we're asking because the decision (keep training vs. wait) genuinely hinges on your plans, and we'd rather build along your intended design than against it. Thanks for both MiniCPM-o and VoxCPM2.

— Luke, nextain (a Korean voice-AI project)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

This is a planning question rather than an implementation task, and it names no files or tests. Start by reviewing the MiniCPM-o 4.5 token-interleaved Talker path and the streaming discussions in #146 and #214. Done means a maintainer confirms whether VoxCPM2 integration is planned and which Korean real-time speech path is supported.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
ai, audio-video-rtc, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.