Training Approach and Strategies for adding Catalan

Hello!

I am trying to teach VoxCPM2 Catalan language. My end goal is to create a Base-type model that may succesfully clone other Catalan voices and speaks the language without the heavy Spanish accent it introduces as-is.

I have developed the following dataset and training conditions:

45 000 clips of multiple speakers (duration ranging between 2 and 29 seconds, with avg of 8.8s) :
90% of which is Catalan, obtained from https://huggingface.co/datasets/projecte-aina/parlament_parla
10% English, obtained from peoples_speech. Adding this English data aims to reduce the risk of catastrophic forgetting.

The structure is {"audio", "text", "ref_audio"}. ref_audio exists in 100% of the rows and I made sure, thanks to the speaker id tag in the parlament_parla dataset that every audio is referenced against one random other audio of the same speaker.

I am trying to run SFT on this dataset only to find that the Catalan pronunciation keeps being overridden by Spanish-like pronunciation. These are my training parameters:

train_sft: batch_size: 2
lr: 0.00002
num_epochs: 5
num_workers: 16
clip_grad_norm: 1.0
gradient_accumulation_steps: 16
mixed_precision: bf16 # "no" | "fp16" | "bf16"
num_processes: 2
weight_decay: 0.0001
sub_talker_loss_weight: 0.2
freeze_code_predictor: true

I would very much appreciate the help to get out of this block, there must be something I am missing, either with my dataset config or my training parameters. Should I introduce other languages, such as Spanish, French or Italian instead of English?

Another question would be, anyone have experience with cascading stage training? I see in this discussion from VibeVoice somebody had sucess with this strategy when implementing an Arabic dialect:
https://huggingface.co/Mohcinimohamed/vibevoice-asr-darija-lora

My two options right now are:

Strategy 1:
SFT in wide, Catalan + English dataset with 90/10 distribution. optionally, Spanish or other romance langs.

Strategy 2:
Three-Stage LoRa training, with a varying rate of Catalan presence in the dataset
Round 1: 70/30 Spanish/Catalan
Round 2: 30/70 Spanish/Catalan
Round 3: 0/100 Spanish/Catalan

Let me know if you have tried any of these strategies and what dataset sizes and training epochs you recommend for teaching this amazing model a new language.

Start with the train_sft configuration and the {"audio", "text", "ref_audio"} dataset structure described in the issue. Compare the proposed SFT and staged LoRA strategies, then evaluate whether a resulting Base-type model can clone Catalan voices without the reported Spanish-like pronunciation.