RVC-Project / RVC-Project/Retrieval-based-Voice-Conversion-WebUI

rvc导出的onnx模型做实时变声推理,声音不连续

Open
#2,277 12 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
38.4k
Forks
5.3k
PR merge metrics
No merged PRs in 30d

Description

环境:
RVC版本:https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI.git -- RVC的git仓库最新提交版本
使用以下脚本导出的onnx:https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI/blob/main/tools/export_onnx.py
原模型为rvc自带的keruanV1.pth,导出模型为keruanV1.onnx,确认生成的是动态模型。

现象:
由于是做实时变声推理,因此每次送入模型的数据片段会很短,大概7200个采样。但输出的数据片段并不能很好的衔接在一起。如:片段A、B、C,在从麦克风进来后,三个数据片段的声音是连续的,但从变声器出来后,听感上A会高一些,B会低一些,而且三个片段的衔接处也无法丝滑过渡,会导致衔接处播放时产生噪音。实际听感就是,变声后的声音会周期性出现“点点点”的声音。

所做的尝试:

  1. 参考https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI/blob/main/gui_v1.py中对推理后数据的处理。增加了交叉淡化算法,但收效甚微。
// 计算当前返回帧的头部淡化效果
auto fo_head = in_buf + sola_offset;
auto fo_tail = fo_head + block_frame;
for (size_t i = 0; i < sola_buffer_frame; i++) {
    fo_head[i] *= fade_in_window[i];
    fo_head[i] += (sola_buffer[i] * fade_out_window[i]);
}
memcpy(sola_buffer, fo_tail, sola_buffer_frame * sizeof(float_t));
  1. 利用ffmpeg的acrossfade滤镜进行处理,但得到的结果一样,效果不大。
  2. 增加每次送入模型的数据量,通过滑动窗口的方式返回数据。如:我会缓存500ms左右的历史变声前数据,新的7200个采样到来后会从缓存尾部推入,缓存头部则会被挤出7200个采样,将整个缓存数据送入变声器。取输出数据尾部的7200个采样返回。
  3. 将3中返回窗口分别设置在输出数据的头部、中部、尾部,均未得到特别好的效果,也将这种窗口形式与1/2中的交叉淡化结合,亦无明显效果。

请问是否有什么办法,能让每次推理的数据片段能衔接起来,让声音连续呢?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading tools/export_onnx.py and the post-inference handling in gui_v1.py. Reproduce the issue with 7200-sample microphone chunks and compare adjacent output chunks, including the attempted crossfade and sliding-window approaches described here. Done means identifying a reproducible cause and a validated method that produces continuous output without boundary noise.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
audio-video-rtc, machine-learning
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.