your checkpoint love 'VoxCPM2Tokenizer' but ONLY 'LlamaTokenizerFast' is present.
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 37.8k
- Forks
- 4.3k
- Avg merge
- 7m
- Merged PRs (30d)
- 1
Description
pip install voxcpm==2.0.2
在调用 VoxCPM.from_pretrained(local_model_path) 时输出警告信息:
The tokenizer class you load from this checkpoint is 'VoxCPM2Tokenizer'.
The class this function is called from is 'LlamaTokenizerFast'.
按下面方法修改 .venv\Lib\site-packages\voxcpm\model\voxcpm2.py,可以消除这个警告。
所以请告诉我:号称 Tokenizer-Free 的模型要的 VoxCPM2Tokenizer 在哪里可以获得?开源吗?
from transformers import LlamaTokenizerFast
+ class VoxCPM2Tokenizer(LlamaTokenizerFast):
+ """VoxCPM2 Tokenizer"""
VoxCPM2Tokenizer
def from_local(cls, path: str, optimize: bool = True, training: bool = False, lora_config: LoRAConfig = None):
config = VoxCPMConfig.model_validate_json(open(os.path.join(path, "config.json")).read())
- tokenizer = LlamaTokenizerFast.from_pretrained(path)
+ tokenizer = VoxCPM2Tokenizer.from_pretrained(path)
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with voxcpm/model/voxcpm2.py and the from_local entry point, then inspect how the checkpoint identifies its tokenizer versus the class loaded by the package. Check the package and checkpoint contents to determine whether VoxCPM2Tokenizer is provided; done means documenting its source or confirming the required loading change.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100