Lightning-AI / Lightning-AI/litgpt
chatting with mistral generates answer with no spaces
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 13.7k
- Forks
- 1.5k
- Avg merge
- 15h 37m
- Merged PRs (30d)
- 1
Description
### Bug description
steps to reproduce the issue:
1. litgpt chat checkpoints/mistralai/Mistral-7B-Instruct-v0.3 --max_new_tokens 2048
2. enter any prompt
I get answer with no space between words, i.e.: "Hello!I'mjustacomputerprogram,soIdon'thavefeelingslikeahumandoes.ButI'mheretohelpyouwithanyquestionsortasksyoumighthave!HowcanIassistyoutoday?"
Here is the full log:
```
⚡ ~ litgpt chat checkpoints/mistralai/Mistral-7B-Instruct-v0.3 --max_new_tokens 2048
{'access_token': None,
'checkpoint_dir': PosixPath('checkpoints/mistralai/Mistral-7B-Instruct-v0.3'),
'compile': False,
'max_new_tokens': 2048,
'multiline': False,
'precision': None,
'quantize': None,
'temperature': 0.8,
'top_k': 50,
'top_p': 1.0}
Now chatting with Mistral-7B-Instruct-v0.3.
To exit, press 'Enter' on an empty prompt.
Seed set to 1234
>> Prompt: hi, how are you?
>> Reply: Hello!I'mjustacomputerprogram,soIdon'thavefeelingslikeahumandoes.ButI'mheretohelpyouwithanyquestionsortasksyoumighthave!HowcanIassistyoutoday?
Time for inference: 3.55 sec total, 12.96 tokens/sec, 46 tokens
```
### What operating system are you using?
Unknown
### LitGPT Version
```
⚡ ~ pip show litgpt | grep Version
Version: 0.5.2
```
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the issue with the mentioned `litgpt chat checkpoints/mistralai/Mistral-7B-Instruct-v0.3 --max_new_tokens 2048` command on LitGPT 0.5.2. Trace the chat output path and token decoding behavior, then verify that the generated reply preserves spaces between words.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100