RVC-Project / RVC-Project/Retrieval-based-Voice-Conversion-WebUI
[Question] Pitch during training
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 38.4k
- Forks
- 5.3k
- PR merge metrics
- No merged PRs in 30d
Description
Hello,
I am looking to modify the model a bit and had a few questions if anyone can shed some light on it :)
Why is the pitch added to the text encoder (enc_p) as well as the GeneratorNSF (dec) during training?
Wouldn't only the GeneratorNSF be what creates the output sound and therefore need the pitch to guide it?
I'm also curious why the text encoder is used in inference, according to the VITS paper it uses the posterior encoder (enc_q) in voice conversion rather than the text encoder (prior encoder/enc_p). Any insight here would be very helpful :D
Many thanks in advance!
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the training and inference paths around the text encoder (enc_p), GeneratorNSF (dec), and posterior encoder (enc_q). Compare their roles with the VITS paper and determine whether the issue can be answered from the current model flow; done means providing a clear explanation of pitch conditioning and inference encoder selection.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- audio-video-rtc, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 15/100