OpenBMB / OpenBMB/MiniCPM-V

[MiniCPM-o-4.5] tts_bound selects a history turn's <|tts_eos|> when current speech is truncated, yielding an empty TTS span

Open
#1,141 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
26.4k
Forks
2.1k
Avg merge
14h 39m
Merged PRs (30d)
3

Description

Hi! I hit an edge case in modeling_minicpmo.py of openbmb/MiniCPM-o-4_5 while testing multi-turn speech chat.

chat() takes the last <|tts_bos|> but the globally last <|tts_eos|> over prompt + generated tokens:

tts_bos_idx = tts_bos_indices[-1] if tts_bos_indices else -1
tts_eos_idx = tts_eos_indices[-1] if tts_eos_indices else None

If the history contains a finished speech turn (prompt has an old <|tts_eos|>) and the current reply stops before its own marker (e.g. a small max_new_tokens), the end index lands before the start index, full_sequences[0][start:end] is empty, and the talker gets an empty condition — no audio, even though the turn had speakable tokens. The bare except around _generate_speech_non_streaming hides the failure, so you just get text with no wav. The None fallback only covers "no <|tts_eos|> anywhere"; it can't help when an old marker sits before the last <|tts_bos|>.

The text-side extraction in the same file already does this right (text.split("<|tts_bos|>")[-1].split("<|tts_eos|>")[0]), so maybe just scope the token path the same way:

start = tts_bos_indices[-1] + 1
eos_in_segment = [i for i in tts_eos_indices if i >= start]
tts_eos_idx = eos_in_segment[0] if eos_in_segment else None
tts_bound = (start, tts_eos_idx)

Single-turn and completed-turn behavior stay unchanged.

(One more thing, probably a separate issue: truncated turns also hit an off-by-one between the token span and the hidden states. In _generate_speech_non_streaming (hidden_text_merge path), llm_tokens is sliced from full_sequences, but hidden_embeds comes from the per-step hidden stack, which only covers positions consumed by a forward pass — the last generated token of a truncated turn is never fed back in, so it has no hidden row, and llm_embeds + hidden_embeds would raises RuntimeError (swallowed by the bare except). Completed turns don't hit this because the tokens generated after the span, like <|tts_eos|>, consume the last span token as input.)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in modeling_minicpmo.py at chat(), where tts_bound is derived, then trace _generate_speech_non_streaming and its hidden_text_merge path. Compare token-span handling with the existing text extraction and inspect available speech-generation tests. Done means truncated multi-turn replies select the current span, produce audio, and avoid a hidden-state length error while completed turns remain unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.