Is VoxCPM2 planned for MiniCPM-o? — and if not, the path to real-time Korean speech
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 37.8k
- Forks
- 4.3k
- Avg merge
- 7m
- Merged PRs (30d)
- 1
Description
We've been building a Korean real-time voice assistant on MiniCPM-o 4.5. We chose MiniCPM-o specifically for its live full-duplex behavior: it produces speech token-interleaved as it generates, so it can start talking immediately and be interrupted mid-sentence. That real-time feel is the core of what we're building — not a nice-to-have.
MiniCPM-o's Korean speech wasn't there out of the box, so we fine-tuned its native Talker for Korean. This has been real, ongoing work and it was paying off — our Korean Talker CER went from ~0.33 to ~0.14 and was still improving.
Then VoxCPM2 came out. Its Korean quality with reference-voice cloning is excellent, and since it's also yours, we tried it as MiniCPM-o's voice. As standalone TTS it's great — but it's batch/utterance synthesis. Inside MiniCPM-o's token-interleaved loop it either comes out disjoint (each small fragment synthesized with its own terminal prosody), or, if we buffer whole clauses first, it adds several seconds of silence before speech and barge-in is gone. (We've seen the earlier streaming issues, e.g. #146 / #214 — our question is specifically about VoxCPM2 as the voice of a token-interleaved full-duplex LLM, not standalone streaming quality.) So VoxCPM2 as the live voice loses exactly the real-time property we built everything around.
That leaves us at a real fork, and only you can answer the first part:
- Is there a plan to bring VoxCPM2 into MiniCPM-o (as its voice, or a streaming integration)? If something official is coming, we'd rather wait for it than keep pushing our own Talker training in parallel.
- If there's no such plan — is there a way to get real-time speech within MiniCPM-o itself that we're missing (a streaming path for VoxCPM2-level quality, or improving the native vocoder)?
- If neither — is the path we're on the right one in your view: continuing to fine-tune the native Korean Talker, plus improving the Thinker for Korean?
We've done the real work here and can share detail and metrics — we're asking because the decision (keep training vs. wait) genuinely hinges on your plans, and we'd rather build along your intended design than against it. Thanks for both MiniCPM-o and VoxCPM2.
— Luke, nextain (a Korean voice-AI project)
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
This is a planning question rather than an implementation task, and it names no files or tests. Start by reviewing the MiniCPM-o 4.5 token-interleaved Talker path and the streaming discussions in #146 and #214. Done means a maintainer confirms whether VoxCPM2 integration is planned and which Korean real-time speech path is supported.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- ai, audio-video-rtc, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100