自己训练hifigan声码器何时能结束？

日志如下：
C:\ProgramData\Anaconda3\envs\mockingbird\python.exe E:\workspace\MockingBird\control\cli\vocoder_train.py my_run e:\datasets hifigan -m E:\workspace\MockingBird\data\ckpt\vocoder\saved_models
Arguments:
run_id: my_run
vocoder_type: hifigan
syn_dir: e:\datasets\SV2TTS\synthesizer
voc_dir: e:\datasets\SV2TTS\vocoder
models_dir: E:\workspace\MockingBird\data\ckpt\vocoder\saved_models
ground_truth: False
save_every: 1000
backup_every: 25000
force_restart: False
config: models/vocoder/hifigan/config_16k_.json

Generator(
(conv_pre): Conv1d(80, 512, kernel_size=(7,), stride=(1,), padding=(3,))
(ups): ModuleList(
(0): ConvTranspose1d(512, 256, kernel_size=(10,), stride=(5,), padding=(3,), output_padding=(1,))
(1): ConvTranspose1d(256, 128, kernel_size=(10,), stride=(5,), padding=(3,), output_padding=(1,))
(2): ConvTranspose1d(128, 64, kernel_size=(8,), stride=(4,), padding=(2,))
(3): ConvTranspose1d(64, 32, kernel_size=(4,), stride=(2,), padding=(1,))
)
(resblocks): ModuleList(
(0): ResBlock1(
(convs1): ModuleList(
(0): Conv1d(256, 256, kernel_size=(3,), stride=(1,), padding=(1,))
(1): Conv1d(256, 256, kernel_size=(3,), stride=(1,), padding=(3,), dilation=(3,))
(2): Conv1d(256, 256, kernel_size=(3,), stride=(1,), padding=(5,), dilation=(5,))
)
(convs2): ModuleList(
(0-2): 3 x Conv1d(256, 256, kernel_size=(3,), stride=(1,), padding=(1,))
)
)
(1): ResBlock1(
(convs1): ModuleList(
(0): Conv1d(256, 256, kernel_size=(7,), stride=(1,), padding=(3,))
(1): Conv1d(256, 256, kernel_size=(7,), stride=(1,), padding=(9,), dilation=(3,))
(2): Conv1d(256, 256, kernel_size=(7,), stride=(1,), padding=(15,), dilation=(5,))
)
(convs2): ModuleList(
(0-2): 3 x Conv1d(256, 256, kernel_size=(7,), stride=(1,), padding=(3,))
)
)
(2): ResBlock1(
(convs1): ModuleList(
(0): Conv1d(256, 256, kernel_size=(11,), stride=(1,), padding=(5,))
(1): Conv1d(256, 256, kernel_size=(11,), stride=(1,), padding=(15,), dilation=(3,))
(2): Conv1d(256, 256, kernel_size=(11,), stride=(1,), padding=(25,), dilation=(5,))
)
(convs2): ModuleList(
(0-2): 3 x Conv1d(256, 256, kernel_size=(11,), stride=(1,), padding=(5,))
)
)
(3): ResBlock1(
(convs1): ModuleList(
(0): Conv1d(128, 128, kernel_size=(3,), stride=(1,), padding=(1,))
(1): Conv1d(128, 128, kernel_size=(3,), stride=(1,), padding=(3,), dilation=(3,))
(2): Conv1d(128, 128, kernel_size=(3,), stride=(1,), padding=(5,), dilation=(5,))
)
(convs2): ModuleList(
(0-2): 3 x Conv1d(128, 128, kernel_size=(3,), stride=(1,), padding=(1,))
)
)
(4): ResBlock1(
(convs1): ModuleList(
(0): Conv1d(128, 128, kernel_size=(7,), stride=(1,), padding=(3,))
(1): Conv1d(128, 128, kernel_size=(7,), stride=(1,), padding=(9,), dilation=(3,))
(2): Conv1d(128, 128, kernel_size=(7,), stride=(1,), padding=(15,), dilation=(5,))
)
(convs2): ModuleList(
(0-2): 3 x Conv1d(128, 128, kernel_size=(7,), stride=(1,), padding=(3,))
)
)
(5): ResBlock1(
(convs1): ModuleList(
(0): Conv1d(128, 128, kernel_size=(11,), stride=(1,), padding=(5,))
(1): Conv1d(128, 128, kernel_size=(11,), stride=(1,), padding=(15,), dilation=(3,))
(2): Conv1d(128, 128, kernel_size=(11,), stride=(1,), padding=(25,), dilation=(5,))
)
(convs2): ModuleList(
(0-2): 3 x Conv1d(128, 128, kernel_size=(11,), stride=(1,), padding=(5,))
)
)
(6): ResBlock1(
(convs1): ModuleList(
(0): Conv1d(64, 64, kernel_size=(3,), stride=(1,), padding=(1,))
(1): Conv1d(64, 64, kernel_size=(3,), stride=(1,), padding=(3,), dilation=(3,))
(2): Conv1d(64, 64, kernel_size=(3,), stride=(1,), padding=(5,), dilation=(5,))
)
(convs2): ModuleList(
(0-2): 3 x Conv1d(64, 64, kernel_size=(3,), stride=(1,), padding=(1,))
)
)
(7): ResBlock1(
(convs1): ModuleList(
(0): Conv1d(64, 64, kernel_size=(7,), stride=(1,), padding=(3,))
(1): Conv1d(64, 64, kernel_size=(7,), stride=(1,), padding=(9,), dilation=(3,))
(2): Conv1d(64, 64, kernel_size=(7,), stride=(1,), padding=(15,), dilation=(5,))
)
(convs2): ModuleList(
(0-2): 3 x Conv1d(64, 64, kernel_size=(7,), stride=(1,), padding=(3,))
)
)
(8): ResBlock1(
(convs1): ModuleList(
(0): Conv1d(64, 64, kernel_size=(11,), stride=(1,), padding=(5,))
(1): Conv1d(64, 64, kernel_size=(11,), stride=(1,), padding=(15,), dilation=(3,))
(2): Conv1d(64, 64, kernel_size=(11,), stride=(1,), padding=(25,), dilation=(5,))
)
(convs2): ModuleList(
(0-2): 3 x Conv1d(64, 64, kernel_size=(11,), stride=(1,), padding=(5,))
)
)
(9): ResBlock1(
(convs1): ModuleList(
(0): Conv1d(32, 32, kernel_size=(3,), stride=(1,), padding=(1,))
(1): Conv1d(32, 32, kernel_size=(3,), stride=(1,), padding=(3,), dilation=(3,))
(2): Conv1d(32, 32, kernel_size=(3,), stride=(1,), padding=(5,), dilation=(5,))
)
(convs2): ModuleList(
(0-2): 3 x Conv1d(32, 32, kernel_size=(3,), stride=(1,), padding=(1,))
)
)
(10): ResBlock1(
(convs1): ModuleList(
(0): Conv1d(32, 32, kernel_size=(7,), stride=(1,), padding=(3,))
(1): Conv1d(32, 32, kernel_size=(7,), stride=(1,), padding=(9,), dilation=(3,))
(2): Conv1d(32, 32, kernel_size=(7,), stride=(1,), padding=(15,), dilation=(5,))
)
(convs2): ModuleList(
(0-2): 3 x Conv1d(32, 32, kernel_size=(7,), stride=(1,), padding=(3,))
)
)
(11): ResBlock1(
(convs1): ModuleList(
(0): Conv1d(32, 32, kernel_size=(11,), stride=(1,), padding=(5,))
(1): Conv1d(32, 32, kernel_size=(11,), stride=(1,), padding=(15,), dilation=(3,))
(2): Conv1d(32, 32, kernel_size=(11,), stride=(1,), padding=(25,), dilation=(5,))
)
(convs2): ModuleList(
(0-2): 3 x Conv1d(32, 32, kernel_size=(11,), stride=(1,), padding=(5,))
)
)
)
(conv_post): Conv1d(32, 1, kernel_size=(7,), stride=(1,), padding=(3,))
)
checkpoints directory : E:\workspace\MockingBird\data\ckpt\vocoder\saved_models\my_run_hifigan
Epoch: 1
C:\Users\Administrator\AppData\Roaming\Python\Python39\site-packages\torch\functional.py:641: UserWarning: stft with return_complex=False is deprecated. In a future pytorch release, stft will return complex tensors for all inputs, and return_complex=False will raise an error.
Note: you can still call torch.view_as_real on the complex output to recover the old return format. (Triggered internally at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\SpectralOps.cpp:867.)
return _VF.stft(input, n_fft, hop_length, win_length, window, # type: ignore[attr-defined]
C:\Users\Administrator\AppData\Roaming\Python\Python39\site-packages\torch\functional.py:641: UserWarning: stft with return_complex=False is deprecated. In a future pytorch release, stft will return complex tensors for all inputs, and return_complex=False will raise an error.
Note: you can still call torch.view_as_real on the complex output to recover the old return format. (Triggered internally at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\SpectralOps.cpp:867.)
return _VF.stft(input, n_fft, hop_length, win_length, window, # type: ignore[attr-defined]
C:\Users\Administrator\AppData\Roaming\Python\Python39\site-packages\torch\functional.py:641: UserWarning: stft with return_complex=False is deprecated. In a future pytorch release, stft will return complex tensors for all inputs, and return_complex=False will raise an error.
Note: you can still call torch.view_as_real on the complex output to recover the old return format. (Triggered internally at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\SpectralOps.cpp:867.)
return _VF.stft(input, n_fft, hop_length, win_length, window, # type: ignore[attr-defined]
C:\Users\Administrator\AppData\Roaming\Python\Python39\site-packages\torch\functional.py:641: UserWarning: stft with return_complex=False is deprecated. In a future pytorch release, stft will return complex tensors for all inputs, and return_complex=False will raise an error.
Note: you can still call torch.view_as_real on the complex output to recover the old return format. (Triggered internally at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\SpectralOps.cpp:867.)
return _VF.stft(input, n_fft, hop_length, win_length, window, # type: ignore[attr-defined]
C:\Users\Administrator\AppData\Roaming\Python\Python39\site-packages\torch\functional.py:641: UserWarning: stft with return_complex=False is deprecated. In a future pytorch release, stft will return complex tensors for all inputs, and return_complex=False will raise an error.
Note: you can still call torch.view_as_real on the complex output to recover the old return format. (Triggered internally at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\SpectralOps.cpp:867.)
return _VF.stft(input, n_fft, hop_length, win_length, window, # type: ignore[attr-defined]
Steps : 0, Gen Loss Total : 108.894, Mel-Spec. Error : 2.420, s/b : 4.006
Saving checkpoint to E:\workspace\MockingBird\data\ckpt\vocoder\saved_models\my_run_hifigan/g_hifigan.pt
Complete.
Saving checkpoint to E:\workspace\MockingBird\data\ckpt\vocoder\saved_models\my_run_hifigan/do_hifigan.pt
Complete.
C:\Users\Administrator\AppData\Roaming\Python\Python39\site-packages\torch\functional.py:641: UserWarning: stft with return_complex=False is deprecated. In a future pytorch release, stft will return complex tensors for all inputs, and return_complex=False will raise an error.
Note: you can still call torch.view_as_real on the complex output to recover the old return format. (Triggered internally at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\SpectralOps.cpp:867.)
return _VF.stft(input, n_fft, hop_length, win_length, window, # type: ignore[attr-defined]
min value is tensor(-1.0063)
max value is tensor(1.0012)
min value is tensor(-1.0209)
max value is tensor(1.0101)
min value is tensor(-1.0140)
max value is tensor(1.0183)
max value is tensor(1.0032)
min value is tensor(-1.0050)
min value is tensor(-1.0193)
min value is tensor(-1.0559)
min value is tensor(-1.0963)
max value is tensor(1.0976)
Steps : 5, Gen Loss Total : 89.431, Mel-Spec. Error : 1.926, s/b : 1.124
Steps : 10, Gen Loss Total : 81.086, Mel-Spec. Error : 1.735, s/b : 1.120
Steps : 15, Gen Loss Total : 78.217, Mel-Spec. Error : 1.641, s/b : 1.121
Steps : 20, Gen Loss Total : 88.838, Mel-Spec. Error : 1.844, s/b : 1.166
Steps : 25, Gen Loss Total : 91.468, Mel-Spec. Error : 1.795, s/b : 1.123
min value is tensor(-1.0038)
Steps : 30, Gen Loss Total : 81.358, Mel-Spec. Error : 1.619, s/b : 1.123
Steps : 35, Gen Loss Total : 81.096, Mel-Spec. Error : 1.561, s/b : 1.166
Steps : 40, Gen Loss Total : 85.270, Mel-Spec. Error : 1.656, s/b : 1.156
Steps : 45, Gen Loss Total : 85.788, Mel-Spec. Error : 1.671, s/b : 1.159
Steps : 50, Gen Loss Total : 81.539, Mel-Spec. Error : 1.538, s/b : 1.117
min value is tensor(-1.0265)
max value is tensor(1.0530)
Steps : 55, Gen Loss Total : 78.400, Mel-Spec. Error : 1.483, s/b : 1.122
min value is tensor(-1.0625)
max value is tensor(1.0293)
Steps : 1000, Gen Loss Total : 76.684, Mel-Spec. Error : 1.461, s/b : 1.154
Saving checkpoint to E:\workspace\MockingBird\data\ckpt\vocoder\saved_models\my_run_hifigan/g_hifigan.pt
Complete.
Saving checkpoint to E:\workspace\MockingBird\data\ckpt\vocoder\saved_models\my_run_hifigan/do_hifigan.pt
Complete.
C:\Users\Administrator\AppData\Roaming\Python\Python39\site-packages\torch\functional.py:641: UserWarning: stft with return_complex=False is deprecated. In a future pytorch release, stft will return complex tensors for all inputs, and return_complex=False will raise an error.
Note: you can still call torch.view_as_real on the complex output to recover the old return format. (Triggered internally at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\SpectralOps.cpp:867.)
return _VF.stft(input, n_fft, hop_length, win_length, window, # type: ignore[attr-defined]
Steps : 1380, Gen Loss Total : 77.188, Mel-Spec. Error : 1.468, s/b : 1.160
Steps : 1385, Gen Loss Total : 75.907, Mel-Spec. Error : 1.452, s/b : 1.161
Steps : 1390, Gen Loss Total : 70.489, Mel-Spec. Error : 1.303, s/b : 1.159
Steps : 1395, Gen Loss Total : 70.901, Mel-Spec. Error : 1.378, s/b : 1.162
Steps : 1400, Gen Loss Total : 69.194, Mel-Spec. Error : 1.249, s/b : 1.175
中间删除掉了一些Steps日志

Start by reading control/cli/vocoder_train.py and models/vocoder/hifigan/config_16k_.json, then reproduce the provided training command and inspect how progress and termination are reported. The issue does not state a code or documentation change, a target training duration, or a completion criterion, so the desired outcome must be clarified before work can be considered done.

Written by the indexing model from the issue text.