[MiniCPM-o-4.5] tts_bound selects a history turn's <|tts_eos|> when current speech is truncated, yielding an empty TTS span
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 26.4k
- Forks
- 2.1k
- Avg merge
- 14h 39m
- Merged PRs (30d)
- 3
Description
Hi! I hit an edge case in modeling_minicpmo.py of openbmb/MiniCPM-o-4_5 while testing multi-turn speech chat.
chat() takes the last <|tts_bos|> but the globally last <|tts_eos|> over prompt + generated tokens:
tts_bos_idx = tts_bos_indices[-1] if tts_bos_indices else -1
tts_eos_idx = tts_eos_indices[-1] if tts_eos_indices else None
If the history contains a finished speech turn (prompt has an old <|tts_eos|>) and the current reply stops before its own marker (e.g. a small max_new_tokens), the end index lands before the start index, full_sequences[0][start:end] is empty, and the talker gets an empty condition — no audio, even though the turn had speakable tokens. The bare except around _generate_speech_non_streaming hides the failure, so you just get text with no wav. The None fallback only covers "no <|tts_eos|> anywhere"; it can't help when an old marker sits before the last <|tts_bos|>.
The text-side extraction in the same file already does this right (text.split("<|tts_bos|>")[-1].split("<|tts_eos|>")[0]), so maybe just scope the token path the same way:
start = tts_bos_indices[-1] + 1
eos_in_segment = [i for i in tts_eos_indices if i >= start]
tts_eos_idx = eos_in_segment[0] if eos_in_segment else None
tts_bound = (start, tts_eos_idx)
Single-turn and completed-turn behavior stay unchanged.
(One more thing, probably a separate issue: truncated turns also hit an off-by-one between the token span and the hidden states. In _generate_speech_non_streaming (hidden_text_merge path), llm_tokens is sliced from full_sequences, but hidden_embeds comes from the per-step hidden stack, which only covers positions consumed by a forward pass — the last generated token of a truncated turn is never fed back in, so it has no hidden row, and llm_embeds + hidden_embeds would raises RuntimeError (swallowed by the bare except). Completed turns don't hit this because the tokens generated after the span, like <|tts_eos|>, consume the last span token as input.)
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in modeling_minicpmo.py at chat(), where tts_bound is derived, then trace _generate_speech_non_streaming and its hidden_text_merge path. Compare token-span handling with the existing text extraction and inspect available speech-generation tests. Done means truncated multi-turn replies select the current span, produce audio, and avoid a hidden-state length error while completed turns remain unchanged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100