ByteDance-Seed / ByteDance-Seed/Bagel

Mismatch between Bagel model's vocab size (152064) and Qwen2 tokenizer's vocab size (151644) causing initialization errors with custom tokens

Open
#265 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
6.2k
Forks
545
PR merge metrics
No merged PRs in 30d

Description

## Content

### Issue Description
I've encountered a discrepancy between the vocabulary sizes in the Bagel model weights and the Qwen2 tokenizer that causes initialization errors when adding custom tokens.

### Details

#### Observed Behavior
- The Bagel model weights have a vocabulary size of 152064:
```python
llm = Qwen2ForCausalLM(llm_config)
print(f"llm.embed_tokens.weight.shape: {llm.model.embed_tokens.weight.shape}") # 152064×3584
print(f"llm.lm_head.weight.shape: {llm.lm_head.weight.shape}") # 152064×3584
```

- However, the Qwen2 tokenizer provided with Bagel has a vocabulary size of only 151644.

#### Issue Context
- When using the original Bagel training pipeline without adding new tokens, this mismatch doesn't cause problems because the code path that would resize the embedding layer isn't triggered.

- However, when I added custom tokens for mesh generation flags, the model initialization fails due to this mismatch. The resize operation attempts to reconcile the tokenizer and model dimensions but encounters the inconsistency.

Any guidance on resolving this inconsistency would be greatly appreciated.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the Bagel initialization path around Qwen2ForCausalLM, llm_config, the tokenizer, and the embedding resize operation when custom tokens are added. Compare the model weight vocabulary size with the tokenizer vocabulary and identify the expected behavior for custom-token initialization. Done means the initialization completes without a dimension mismatch while preserving the intended token and embedding sizes.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.