IndexError and UnpicklingError during model training with pretrained weights

When trying to train I encountered two errors: an IndexError related to the list index being out of range and an _pickle.UnpicklingError regarding an invalid load key.

INFO:mi-test:{'train': {'log_interval': 200, 'seed': 1234, 'epochs': 20000, 'learning_rate': 0.0001, 'betas': [0.8, 0.99], 'eps': 1e-09, 'batch_size': 40, 'fp16_run': True, 'lr_decay': 0.999875, 'segment_size': 12800, 'init_lr_ratio': 1, 'warmup_epochs': 0, 'c_mel': 45, 'c_kl': 1.0}, 'data': {'max_wav_value': 32768.0, 'sampling_rate': 40000, 'filter_length': 2048, 'hop_length': 400, 'win_length': 2048, 'n_mel_channels': 125, 'mel_fmin': 0.0, 'mel_fmax': None, 'training_files': './logs/mi-test/filelist.txt'}, 'model': {'inter_channels': 192, 'hidden_channels': 192, 'filter_channels': 768, 'n_heads': 2, 'n_layers': 6, 'kernel_size': 3, 'p_dropout': 0, 'resblock': '1', 'resblock_kernel_sizes': [3, 7, 11], 'resblock_dilation_sizes': [[1, 3, 5], [1, 3, 5], [1, 3, 5]], 'upsample_rates': [10, 10, 2, 2], 'upsample_initial_channel': 512, 'upsample_kernel_sizes': [16, 16, 4, 4], 'use_spectral_norm': False, 'gin_channels': 256, 'spk_embed_dim': 109}, 'model_dir': './logs/mi-test', 'experiment_dir': './logs/mi-test', 'save_every_epoch': 10, 'name': 'mi-test', 'total_epoch': 200, 'pretrainG': 'pretrained/f0G40k.pth', 'pretrainD': 'pretrained/f0D40k.pth', 'version': 'v1', 'gpus': '0', 'sample_rate': '40k', 'if_f0': 1, 'if_latest': 1, 'save_every_weights': '1', 'if_cache_data_in_gpu': 0}
INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
gin_channels: 256 self.spk_embed_dim: 109
INFO:mi-test:loaded pretrained pretrained/f0G40k.pth pretrained/f0D40k.pth
Process Process-1:
Traceback (most recent call last):
File "/mnt/c/users/john/documents/Retrieval-based-Voice-Conversion-WebUI/train_nsf_sim_cache_sid_load_pretrain.py", line 178, in run
utils.latest_checkpoint_path(hps.model_dir, "D_*.pth"), net_d, optim_d
File "/mnt/c/users/john/documents/Retrieval-based-Voice-Conversion-WebUI/train/utils.py", line 206, in latest_checkpoint_path
x = f_list[-1]
IndexError: list index out of range

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/se7dev/miniconda3/envs/RBVC/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/home/se7dev/miniconda3/envs/RBVC/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/mnt/c/users/john/documents/Retrieval-based-Voice-Conversion-WebUI/train_nsf_sim_cache_sid_load_pretrain.py", line 197, in run
torch.load(hps.pretrainG, map_location="cpu")["model"]
File "/home/se7dev/miniconda3/envs/RBVC/lib/python3.10/site-packages/torch/serialization.py", line 815, in load
return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
File "/home/se7dev/miniconda3/envs/RBVC/lib/python3.10/site-packages/torch/serialization.py", line 1033, in _legacy_load
magic_number = pickle_module.load(f, **pickle_load_args)
_pickle.UnpicklingError: invalid load key, 'v'.

Reproduce the training run from train_nsf_sim_cache_sid_load_pretrain.py using the reported configuration, then inspect train/utils.py:206 and the fallback loading path around lines 178-197. Check how the D_*.pth file list and pretrained/f0G40k.pth are handled; done means training starts without either reported exception.