RVC-Project / RVC-Project/Retrieval-based-Voice-Conversion-WebUI

Training Speed Discrepancy & Torch Issues Resolution

Open
#1,077 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

following up
Dominant language
Python
Stars
38.4k
Forks
5.3k
PR merge metrics
No merged PRs in 30d

Description

The training process in the previous version was twice as fast. Previously, I utilized 'crepe' for training, whereas currently, I'm using 'rmvpe_gpu'.

If I were to use 'crepe' in the current version, each epoch might take around 15 minutes, compared to less than a minute in the past. It's important to note that I'm conducting these comparisons on the same computer, GPU, and identical settings.

In the past, I employed a batch size of 8 on an RTX 3070 graphics card to achieve faster results. Now, to halve the training time, I've switched to a batch size of 4 on the same card.

However, in the current version, I'm encountering the following error: "WARNING | xformers | A matching Triton is not available, some optimizations will not be enabled.
Error caught was: No module named 'triton'."
Could you please provide a solution for this?

Additionally, I'm experiencing Torch-related issues despite multiple uninstallations, reinstalls, and attempts with different versions. The problems persist unchanged. Could you offer guidance on resolving these Torch-related problems?

(((return_complex=False is deprecated. In a future pytorch release, stft will return complex tensors for all inputs, and return_complex=False will raise an error.
Note: you can still call torch.view_as_real on the complex output to recover the old return format. (Triggered internally at C:\actions-runner_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\SpectralOps.cpp:867.)
return _VF.stft(input, n_fft, hop_length, win_length, window, # type: ignore[attr-defined]
F:\RVC-2\RVC0813Nvidia\runtime\lib\site-packages\torch\functional.py:641: UserWarning: stft with return_complex=False is deprecated. In a future pytorch release, stft will return complex tensors for all inputs, and return_complex=False will raise an error.
Note: you can still call torch.view_as_real on the complex output to recover the old return format. (Triggered internally at C:\actions-runner_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\SpectralOps.cpp:867.)
return _VF.stft(input, n_fft, hop_length, win_length, window, # type: ignore[attr-defined]
F:\RVC-2\RVC0813Nvidia\runtime\lib\site-packages\torch\functional.py:641: UserWarning: stft with return_complex=False is deprecated. In a future pytorch release, stft will return complex tensors for all inputs, and return_complex=False will raise an error.
Note: you can still call torch.view_as_real on the complex output to recover the old return format. (Triggered internally at C:\actions-runner_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\SpectralOps.cpp:867.)
return _VF.stft(input, n_fft, hop_length, win_length, window, # type: ignore[attr-defined]
F:\RVC-2\RVC0813Nvidia\runtime\lib\site-packages\torch\functional.py:641: UserWarning: stft with return_complex=False is deprecated. In a future pytorch release, stft will return complex tensors for all inputs, and return_complex=False will raise an error.
Note: you can still call torch.view_as_real on the complex output to recover the old return format. (Triggered internally at C:\actions-runner_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\SpectralOps.cpp:867.)
return _VF.stft(input, n_fft, hop_length, win_length, window, # type: ignore[attr-defined]
F:\RVC-2\RVC0813Nvidia\runtime\lib\site-packages\torch\functional.py:641: UserWarning: stft with return_complex=False is deprecated. In a future pytorch release, stft will return complex tensors for all inputs, and return_complex=False will raise an error.
Note: you can still call torch.view_as_real on the complex output to recover the old return format. (Triggered internally at C:\actions-runner_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\SpectralOps.cpp:867.)
return VF.stft(input, n_fft, hop_length, win_length, window, # type: ignore[attr-defined]
INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
F:\RVC-2\RVC0813Nvidia\runtime\lib\site-packages\torch\autograd_init
.py:200: UserWarning: Grad strides do not match bucket view strides. This may indicate grad was not created according to the gradient layout contract, or that the param's strides changed since DDP was constructed. This is not an error, but may impair performance.))

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No source file or test is named. Start by reproducing the training-speed comparison with crepe and rmvpe_gpu on the stated RTX 3070 settings, then inspect the xformers/Triton message and the reported PyTorch STFT and distributed-training warnings. Done means identifying the cause of the discrepancy and providing a verified resolution for the dependency and Torch problems.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
audio-video-rtc, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.