deepseek-ai / deepseek-ai/DeepSeek-Coder
What's the pad token for deepseek-coder
- Dominant language
- Python
- Stars
- 24.3k
- Forks
- 2.9k
- PR merge metrics
- No merged PRs in 30d
Description
Dear experts,
I found there are two pad tokens in deepseek-coder. What's the difference between them?
When I need to use pad token, which one shall I use?
- tokenizer.json
```json
{
"id": 32018,
"content": "",
"single_word": false,
"lstrip": false,
"rstrip": false,
"normalized": true,
"special": false
},
```
- tokenizer_config.json
```json
"pad_token": {
"__type": "AddedToken",
"content": "<|end▁of▁sentence|>",
"lstrip": false,
"normalized": true,
"rstrip": false,
"single_word": false
},
```
Also, why the second pad token is same as token 32014? I assume it is on purpose. Could you please help to explain the reason?
```json
{
"id": 32013,
"content": "<|begin▁of▁sentence|>",
"single_word": false,
"lstrip": false,
"rstrip": false,
"normalized": true,
"special": true
},
{
"id": 32014,
"content": "<|end▁of▁sentence|>",
"single_word": false,
"lstrip": false,
"rstrip": false,
"normalized": true,
"special": true
},
```
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.