RVC-Project / RVC-Project/Retrieval-based-Voice-Conversion-WebUI
rvc导出的onnx模型做实时变声推理,声音不连续
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 38.4k
- Forks
- 5.3k
- PR merge metrics
- No merged PRs in 30d
Description
环境:
RVC版本:https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI.git -- RVC的git仓库最新提交版本
使用以下脚本导出的onnx:https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI/blob/main/tools/export_onnx.py
原模型为rvc自带的keruanV1.pth,导出模型为keruanV1.onnx,确认生成的是动态模型。
现象:
由于是做实时变声推理,因此每次送入模型的数据片段会很短,大概7200个采样。但输出的数据片段并不能很好的衔接在一起。如:片段A、B、C,在从麦克风进来后,三个数据片段的声音是连续的,但从变声器出来后,听感上A会高一些,B会低一些,而且三个片段的衔接处也无法丝滑过渡,会导致衔接处播放时产生噪音。实际听感就是,变声后的声音会周期性出现“点点点”的声音。
所做的尝试:
- 参考https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI/blob/main/gui_v1.py中对推理后数据的处理。增加了交叉淡化算法,但收效甚微。
// 计算当前返回帧的头部淡化效果
auto fo_head = in_buf + sola_offset;
auto fo_tail = fo_head + block_frame;
for (size_t i = 0; i < sola_buffer_frame; i++) {
fo_head[i] *= fade_in_window[i];
fo_head[i] += (sola_buffer[i] * fade_out_window[i]);
}
memcpy(sola_buffer, fo_tail, sola_buffer_frame * sizeof(float_t));
- 利用ffmpeg的acrossfade滤镜进行处理,但得到的结果一样,效果不大。
- 增加每次送入模型的数据量,通过滑动窗口的方式返回数据。如:我会缓存500ms左右的历史变声前数据,新的7200个采样到来后会从缓存尾部推入,缓存头部则会被挤出7200个采样,将整个缓存数据送入变声器。取输出数据尾部的7200个采样返回。
- 将3中返回窗口分别设置在输出数据的头部、中部、尾部,均未得到特别好的效果,也将这种窗口形式与1/2中的交叉淡化结合,亦无明显效果。
请问是否有什么办法,能让每次推理的数据片段能衔接起来,让声音连续呢?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading tools/export_onnx.py and the post-inference handling in gui_v1.py. Reproduce the issue with 7200-sample microphone chunks and compare adjacent output chunks, including the attempted crossfade and sliding-window approaches described here. Done means identifying a reproducible cause and a validated method that produces continuous output without boundary noise.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- audio-video-rtc, machine-learning
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100