RVC-Project / RVC-Project/Retrieval-based-Voice-Conversion-WebUI
H800训练会使loss异常增加到nan？
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 38.4k
- Forks
- 5.3k
- PR merge metrics
- No merged PRs in 30d
Description
不知道有没有人用H800训练过。
我的个人电脑，windows系统，显卡3060，python3.10, cuda11.7,torch2.3可以正常训练，20几个epoch就可以将初始为31的loss_mel训练到15左右，得到的pth和index推理结果也正常。
计算服务器是linux系统，显卡H800，python3.9 3.10, cuda12.2, torch2.1~2.5都试过，使用1-4块H800都试过，使用和前述同样的数据和配置（基本就是gitclone下来之后没再动过），但训练时loss_mel会从30逐渐升至50，60，之后变为nan。得到的pth和index推理出的结果几乎为纯蜂鸣声。
看了其他问题的回答修改过fp16_run 为false，不起作用；减小learning_rate的初值，虽然在200轮以内不会出现nan，但得到的pth推理出的音频仍然充满电流声与蜂鸣声。
不知道是cuda版本的问题，pytorch版本的问题，还是显卡的问题？有没有用H800出现同样问题的？
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Compare the reported Windows/RTX 3060 environment with the Linux/H800 environments, including Python, CUDA, PyTorch, GPU count, and training settings. Reproduce the training run with the same data and configuration, then determine which environment difference corresponds to rising loss, NaN values, and buzzy inference output.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100