ml-explore / ml-explore/mlx-examples
Llama-3.1-8B-Instruct-4bit keeps looping at the end.
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 9k
- Forks
- 1.2k
- PR merge metrics
- No merged PRs in 30d
Description
I'm on mlx-lm v0.19.1.
Running the following command with 4bit produced a bug where it would just generate the full 1000 max-tokens and just repeat the last two paragraphs over and over.
mlx_lm.generate --model mlx-community/Meta-Llama-3.1-8B-Instruct-4bit --max-kv-size 33000 --max-tokens 1000 --temp 0.0 --top-p 0.9 --seed 1000 --prompt -<../text/portugal.txt;say done
Here is my Full prompt from Wikipedia article.
Just to see what happens, I increased --max-kv-size to 34k and --max-tokens to 2000, and it generated 2k tokens with the loop.
Running the exact same command replacing 4bit with 8bit generated the correct text with 811 tokens and stopped at the end without the loop.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the provided mlx_lm.generate command with the 4bit model and compare its output and stopping behavior with the 8bit command. Trace the generation and stopping behavior for the 4bit path; done means the 4bit model produces the expected text and stops without repeating paragraphs, with coverage for the reported setup.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100