ByteDance-Seed / ByteDance-Seed/Bagel
Mismatch between Bagel model's vocab size (152064) and Qwen2 tokenizer's vocab size (151644) causing initialization errors with custom tokens
- Dominant language
- Python
- Stars
- 6.2k
- Forks
- 545
- PR merge metrics
- No merged PRs in 30d
Description
## Content
### Issue Description
I've encountered a discrepancy between the vocabulary sizes in the Bagel model weights and the Qwen2 tokenizer that causes initialization errors when adding custom tokens.
### Details
#### Observed Behavior
- The Bagel model weights have a vocabulary size of 152064:
```python
llm = Qwen2ForCausalLM(llm_config)
print(f"llm.embed_tokens.weight.shape: {llm.model.embed_tokens.weight.shape}") # 152064×3584
print(f"llm.lm_head.weight.shape: {llm.lm_head.weight.shape}") # 152064×3584
```
- However, the Qwen2 tokenizer provided with Bagel has a vocabulary size of only 151644.
#### Issue Context
- When using the original Bagel training pipeline without adding new tokens, this mismatch doesn't cause problems because the code path that would resize the embedding layer isn't triggered.
- However, when I added custom tokens for mesh generation flags, the model initialization fails due to this mismatch. The resize operation attempts to reconcile the tokenizer and model dimensions but encounters the inconsistency.
Any guidance on resolving this inconsistency would be greatly appreciated.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing the Bagel initialization path around Qwen2ForCausalLM, llm_config, the tokenizer, and the embedding resize operation when custom tokens are added. Compare the model weight vocabulary size with the tokenizer vocabulary and identify the expected behavior for custom-token initialization. Done means the initialization completes without a dimension mismatch while preserving the intended token and embedding sizes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 38/100