deepseek-ai / deepseek-ai/DeepSeek-Coder

deepseek-coder-7b-base-v1.5 tokenizer=LlamaTokenizerFast 为什么 分词会有很多乱码字符呢?

Open
#129 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
24.3k
Forks
2.9k
No merged PRs in 30d

Description

![image](https://github.com/deepseek-ai/DeepSeek-Coder/assets/20754435/53540b6d-7ca6-437f-be72-5c8318182843)
![tokenizer乱码](https://github.com/deepseek-ai/DeepSeek-Coder/assets/20754435/59db7e6c-029a-4bd8-b8f5-09407affb0cd)

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.