Training does not continue, but begins anew.

My training stopped due to a full disk, I cleared the space, entered all the same settings, it says that point 660 is loaded, but it starts anyway from the 1st epoch.

runtime\python.exe train_nsf_sim_cache_sid_load_pretrain.py -e "my-voice3.0" -sr 48k -f0 1 -bs 28 -g 0 -te 1000 -se 20 -pg pretrained_v2/f0G48k.pth -pd pretrained_v2/f0D48k.pth -l 0 -c 1 -sw 1 -v v2
INFO:my-voice3.0:{'train': {'log_interval': 200, 'seed': 1234, 'epochs': 20000, 'learning_rate': 0.0001, 'betas': [0.8, 0.99], 'eps': 1e-09, 'batch_size': 28, 'fp16_run': True, 'lr_decay': 0.999875, 'segment_size': 17280, 'init_lr_ratio': 1, 'warmup_epochs': 0, 'c_mel': 45, 'c_kl': 1.0}, 'data': {'max_wav_value': 32768.0, 'sampling_rate': 48000, 'filter_length': 2048, 'hop_length': 480, 'win_length': 2048, 'n_mel_channels': 128, 'mel_fmin': 0.0, 'mel_fmax': None, 'training_files': './logs\\my-voice3.0/filelist.txt'}, 'model': {'inter_channels': 192, 'hidden_channels': 192, 'filter_channels': 768, 'n_heads': 2, 'n_layers': 6, 'kernel_size': 3, 'p_dropout': 0, 'resblock': '1', 'resblock_kernel_sizes': [3, 7, 11], 'resblock_dilation_sizes': [[1, 3, 5], [1, 3, 5], [1, 3, 5]], 'upsample_rates': [12, 10, 2, 2], 'upsample_initial_channel': 512, 'upsample_kernel_sizes': [24, 20, 4, 4], 'use_spectral_norm': False, 'gin_channels': 256, 'spk_embed_dim': 109}, 'model_dir': './logs\\my-voice3.0', 'experiment_dir': './logs\\my-voice3.0', 'save_every_epoch': 20, 'name': 'my-voice3.0', 'total_epoch': 1000, 'pretrainG': 'pretrained_v2/f0G48k.pth', 'pretrainD': 'pretrained_v2/f0D48k.pth', 'version': 'v2', 'gpus': '0', 'sample_rate': '48k', 'if_f0': 1, 'if_latest': 0, 'save_every_weights': '1', 'if_cache_data_in_gpu': 1}
INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
gin_channels: 256 self.spk_embed_dim: 109
./logs\my-voice3.0\D_31680.pth
INFO:my-voice3.0:Loaded model weights
INFO:my-voice3.0:Loaded checkpoint './logs\my-voice3.0\D_31680.pth' (epoch 660)
INFO:my-voice3.0:loaded D
./logs\my-voice3.0\G_32160.pth
INFO:my-voice3.0:loaded pretrained pretrained_v2/f0G48k.pth
<All keys matched successfully>
INFO:my-voice3.0:loaded pretrained pretrained_v2/f0D48k.pth
<All keys matched successfully>
E:\ai\RVC0813Nvidia\runtime\lib\site-packages\torch\functional.py:641: UserWarning: stft with return_complex=False is deprecated. In a future pytorch release, stft will return complex tensors for all inputs, and return_complex=False will raise an error.
Note: you can still call torch.view_as_real on the complex output to recover the old return format. (Triggered internally at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\SpectralOps.cpp:867.)
  return _VF.stft(input, n_fft, hop_length, win_length, window,  # type: ignore[attr-defined]
INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
E:\ai\RVC0813Nvidia\runtime\lib\site-packages\torch\autograd\__init__.py:200: UserWarning: Grad strides do not match bucket view strides. This may indicate grad was not created according to the gradient layout contract, or that the param's strides changed since DDP was constructed.  This is not an error, but may impair performance.
grad.sizes() = [64, 1, 4], strides() = [4, 1, 1]
bucket_view.sizes() = [64, 1, 4], strides() = [4, 4, 1] (Triggered internally at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\torch\csrc\distributed\c10d\reducer.cpp:337.)
  Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
INFO:my-voice3.0:Train Epoch: 1 [0%]
INFO:my-voice3.0:[0, 0.0001]
INFO:my-voice3.0:loss_disc=3.944, loss_gen=3.080, loss_fm=12.032,loss_mel=27.263, loss_kl=7.555
DEBUG:matplotlib:matplotlib data path: E:\ai\RVC0813Nvidia\runtime\lib\site-packages\matplotlib\mpl-data
DEBUG:matplotlib:CONFIGDIR=C:\Users\remot\.matplotlib
DEBUG:matplotlib:interactive is False
DEBUG:matplotlib:platform is win32
INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
Keyboard interruption in main thread... closing server.
Process Process-1:

Start with train_nsf_sim_cache_sid_load_pretrain.py and trace how the checkpoint paths in logs/my-voice3.0 are loaded and how the starting epoch is initialized. Re-run the supplied command with the reported checkpoints; done means training resumes from the loaded epoch instead of logging Train Epoch: 1.