OpenBMB / OpenBMB/VoxCPM

Asking about the best practice to fine-tuning?

Open
#271 2 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
37.8k
Forks
4.3k
Avg merge
7m
Merged PRs (30d)
1

Description

Goal

mimic a specific timbre

dataset preparation

Question 1: length of all the audio segments combined shall be at least?
Question 2: recommended length range of each segment?
Question 3: Is it beneficial to trim the leading and ending silence? What about leading and ending breath(I have methods to trim it, so I just wonder whether we shall trim it)?
Question 4: Do we need to normalize volume?

hyper-params and success metrics

Question 5: How to tune batch size and maximum steps? Is there an empirical formula for the size of the dataset vs maximum steps?
Question 6: I have noticed a phenomenon during fine-tuning: the loss on validation datasets has risen after a few epochs, but the timbre similarity improves. So do we need to watch the loss on val datasets at all? Is the training loss the only thing we need to watch?
Question 7: what loss/diff is considered to be 'good', which seems to drop unsteadily.

other confusions

Question 8: Without reference audios and only given the lora-fted model, the timbre may vary a lot: the varying can be like emotional changes, volume changes and pitch changes. It is unrealistic that the training clips are all very similar in emotions, tones etc. And we do want some emotional changes when it reads long texts. But the changes are too abrupt and often unintended. What is the best way to increase the steadiness of the trained model?

Many thanks to any helpful suggestions!

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the issue's eight questions about dataset preparation, fine-tuning hyperparameters, validation loss, and timbre stability. Done means providing project-specific, documented guidance addressing these questions; the payload names no files, tests, or entry points to investigate.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Documentation
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.