[Question] streaming generate模式下roleplay生成音频音色不统一
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 26.4k
- Forks
- 2.1k
- Avg merge
- 14h 39m
- Merged PRs (30d)
- 3
Description
在huggingface上作者们给出了使用model.chat在roleplay模式下的生成。我尝试在streaming_generate下使用roleplay模式,但是发现声音并不和我给的ref audio相似,甚至声音的性别也不相似,在多轮对话中声音也不统一。我不确定这是streaming的函数逻辑对prompt要求不同,还是其他原因?具体而言,我的代码逻辑如下:
`
ref_path = ‘my custom audio’
audio_prompt, sr = librosa.load(ref_path, sr=16000, mono=True)
sys_msg = self.minicpmo_model.get_sys_prompt(ref_audio=audio_prompt, mode='audio_roleplay', language='en')
1. prefill system prompt
res = model.streaming_prefill(
session_id=session_id,
msgs=[sys_msg], # 在这里将roleplay模式下的prompt进行prefill
tokenizer=tokenizer
)
2. prefill video/audio chunks
for content in contents:
msgs = [{"role":"user", "content": content}]
res = model.streaming_prefill(
session_id=session_id,
msgs=msgs, # 在这里对输入进行prefill
tokenizer=tokenizer
)
3. generate
res = model.streaming_generate(
session_id=session_id,
tokenizer=tokenizer,
temperature=0.5,
generate_audio=generate_audio
)
后续streaming生成
`
如果这和我的环境有关/我哪步实现有误,我可以提供我的详细脚本,我不确定这是bug还是我自己实现的原因?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the provided flow with get_sys_prompt, streaming_prefill, and streaming_generate, using the custom reference audio and audio_roleplay mode. Compare the streaming behavior with the model.chat roleplay usage described in the issue; done means determining whether the inconsistent voice is an implementation bug or an incorrect streaming usage pattern.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- huggingface, python
- Domain
- ai, audio-video-rtc
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100