OpenBMB / OpenBMB/MiniCPM-V

[Question] streaming generate模式下roleplay生成音频音色不统一

Open
#965 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
26.4k
Forks
2.1k
Avg merge
14h 39m
Merged PRs (30d)
3

Description

在huggingface上作者们给出了使用model.chat在roleplay模式下的生成。我尝试在streaming_generate下使用roleplay模式,但是发现声音并不和我给的ref audio相似,甚至声音的性别也不相似,在多轮对话中声音也不统一。我不确定这是streaming的函数逻辑对prompt要求不同,还是其他原因?具体而言,我的代码逻辑如下:
`
ref_path = ‘my custom audio’
audio_prompt, sr = librosa.load(ref_path, sr=16000, mono=True)
sys_msg = self.minicpmo_model.get_sys_prompt(ref_audio=audio_prompt, mode='audio_roleplay', language='en')

1. prefill system prompt

res = model.streaming_prefill(
session_id=session_id,
msgs=[sys_msg], # 在这里将roleplay模式下的prompt进行prefill
tokenizer=tokenizer
)

2. prefill video/audio chunks

for content in contents:
msgs = [{"role":"user", "content": content}]
res = model.streaming_prefill(
session_id=session_id,
msgs=msgs, # 在这里对输入进行prefill
tokenizer=tokenizer
)

3. generate

res = model.streaming_generate(
session_id=session_id,
tokenizer=tokenizer,
temperature=0.5,
generate_audio=generate_audio
)

后续streaming生成

`

如果这和我的环境有关/我哪步实现有误,我可以提供我的详细脚本,我不确定这是bug还是我自己实现的原因?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the provided flow with get_sys_prompt, streaming_prefill, and streaming_generate, using the custom reference audio and audio_roleplay mode. Compare the streaming behavior with the model.chat roleplay usage described in the issue; done means determining whether the inconsistent voice is an implementation bug or an incorrect streaming usage pattern.

Written by the indexing model from the issue text.

Assessment

Tech stack
huggingface, python
Domain
ai, audio-video-rtc
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.