ml-explore / ml-explore/mlx-examples
GPU Usage dropping before completion ends
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 9k
- Forks
- 1.2k
- PR merge metrics
- No merged PRs in 30d
Description
I have been using the new Command-R+ model in 4-bit mode and consistently observe a drop in GPU utilization immediately after prompt evaluation, as it begins generation/prediction. This leads to significantly reduced performance.
During evaluation:
During generation – drop occurs right before the first token is predicted (i.e. "<PAD>"):
Here's my setup:
Machine: Apple M2 Ultra (cores: 8E+16P+60GPU), 192GB Ram
ProductName: macOS
ProductVersion: 14.3
BuildVersion: 23D56
I have tried with and without setting my memory limit:
sudo sysctl iogpu.wired_lwm_mb=150000
I have tried with and without disabling the cache:
mx.metal.set_cache_limit(0)
Any help would be welcome, because at the moment I am only able to use the llama.cpp implementation of Command-R+, which works without any issues.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue names no source files or tests. Reproduce Command-R+ in 4-bit mode during generation on the stated M2 Ultra setup, comparing GPU utilization before and after prompt evaluation; done when the cause of the pre-first-token drop is identified and a regression check or clearly verified fix is available.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100