Lightning-AI / Lightning-AI/litgpt

Resolve output characters garbled

Open
#1,422 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
13.7k
Forks
1.5k
Avg merge
15h 37m
Merged PRs (30d)
1

Description

hello , Because I want my model to have Chinese language ability, but the language and training resources required for full parameter training of Chinese language are huge, I used the Chinese llama model trained by other open-source projects. However, I think the litgpt project is very convenient, so I converted the models from other open-source projects to a lit model. The output of the lit model has Chinese language ability, but there is character garbled phenomenon in the output text. How can I solve the character garbled phenomenon? I look forward to your reply. Thank you
(Appendix: Chinese Open Source Model GitHub Address: https://github.com/LlamaFamily/Llama-Chinese?tab=readme-ov-file
Chinese Open Source Model File Hugging Face Address: https://huggingface.co/FlagAlpha/Llama3-Chinese-8B-Instruct/tree/main

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the garbled output from the converted lit model and compare it with the referenced Llama-Chinese model and Hugging Face files. Done means identifying the conversion or model-resource mismatch responsible and documenting a reliable way to produce readable Chinese output.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.