OpenBMB / OpenBMB/VoxCPM

Inquiry on "is_prompt" usage in training process

Open
#359 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
37.8k
Forks
4.3k
Avg merge
7m
Merged PRs (30d)
1

Description

Hi VoxCPM Team,

I am exploring the fine-tuning code of VoxCPM and noticed the is_prompt parameter in the dataset processing/collation stage packers.py By default, is_prompt is set to False during fine-tuning. However, looking into the mechanics, flipping this parameter to True changes the structural arrangement of the sequence and the behavior of the loss_mask:

  • When is_prompt=True: The sequence layout is restructured. The reference audio/text segment is explicitly separated and treated as a "prefix" or contextual prompt at the beginning of the sequence, paving the way for the subsequent continuation sequence.
  • When is_prompt=True: The loss_mask for the prompt prefix section is forcefully set to 0. The loss computation and gradient backpropagation are tightly constrained only to the continuation part (the text/audio following the prompt prefix).

Questions & Core Concerns

  1. Is this parameter explicitly designed for Continuous and Ultimate Clone training?
  2. Fine-tuning mix without is_prompt=True: Will it fail to optimize continuation ability?
    Could you confirm if keeping is_prompt=False on data with references will indeed dilute or fail to optimize the targeted continuation/generation capability? What is your recommended data configuration and is_prompt toggling strategy when mixing vanilla TTS data and prefix-conditioned audio data?

Thank you for your time and for open-sourcing this amazing framework! Looking forward to your insights.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with src/voxcpm/training/packers.py and trace how is_prompt changes sequence layout and loss_mask during collation. Compare the false and true paths against the fine-tuning scenarios described in the issue; done means documenting whether each mode supports continuation training and how mixed data should configure is_prompt.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
audio-video-rtc, machine-learning
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.