Inquiry on "is_prompt" usage in training process
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 37.8k
- Forks
- 4.3k
- Avg merge
- 7m
- Merged PRs (30d)
- 1
Description
Hi VoxCPM Team,
I am exploring the fine-tuning code of VoxCPM and noticed the is_prompt parameter in the dataset processing/collation stage packers.py By default, is_prompt is set to False during fine-tuning. However, looking into the mechanics, flipping this parameter to True changes the structural arrangement of the sequence and the behavior of the loss_mask:
- When
is_prompt=True: The sequence layout is restructured. The reference audio/text segment is explicitly separated and treated as a "prefix" or contextual prompt at the beginning of the sequence, paving the way for the subsequent continuation sequence. - When
is_prompt=True: The loss_mask for the prompt prefix section is forcefully set to 0. The loss computation and gradient backpropagation are tightly constrained only to the continuation part (the text/audio following the prompt prefix).
Questions & Core Concerns
- Is this parameter explicitly designed for Continuous and Ultimate Clone training?
- Fine-tuning mix without is_prompt=True: Will it fail to optimize continuation ability?
Could you confirm if keepingis_prompt=Falseon data with references will indeed dilute or fail to optimize the targeted continuation/generation capability? What is your recommended data configuration and is_prompt toggling strategy when mixing vanilla TTS data and prefix-conditioned audio data?
Thank you for your time and for open-sourcing this amazing framework! Looking forward to your insights.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with src/voxcpm/training/packers.py and trace how is_prompt changes sequence layout and loss_mask during collation. Compare the false and true paths against the fine-tuning scenarios described in the issue; done means documenting whether each mode supports continuation training and how mixed data should configure is_prompt.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- audio-video-rtc, machine-learning
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100