EveryVoiceTTS / EveryVoiceTTS/EveryVoice

When doing a FP fine-tune , in Tensorboard the next round start 50 steps ahead.

Open
#550 2 comments 0 reactions 1 assignee Claimed by @SamuelLarkin View on GitHub
documentation
Dominant language
Python
Stars
45
Forks
4
Avg merge
1d 2h
Merged PRs (30d)
14

Description

### Bug description

When doing a FP fine-tune , in Tensorboard it looks like the next round start 50 steps ahead.

See image
![Screenshot 2024-09-18 at 09 59 35](https://github.com/user-attachments/assets/aea8bf28-688e-4118-94e5-0bc881c6de3e)

### How to reproduce the bug

```python
Run a FP training . One epoch is good enough to see.

srun everyvoice train text-to-spec config/everyvoice-text-to-spec.yaml --config-args training.max_epochs=1

Then fine-fine that job with an extra epoch.

srun everyvoice train text-to-spec config/everyvoice-text-to-spec.yaml --config-args training.max_epochs=2 --config-args training.finetune_checkpoint="logs_and_checkpoints/FeaturePredictionExperiment/base/checkpoints/last.ckpt"
```

### Error messages and logs

No error message.

### Environment

Standard ENV , nothing special. This will be for after PR 547 is merged. ( Resume at the end of the last trained epoch #547
, Issue #534 )

### More info

none

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.