ml-explore / ml-explore/mlx-examples
Keep `dtype` of Models
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 9k
- Forks
- 1.2k
- PR merge metrics
- No merged PRs in 30d
Description
While playing around with the TinyLlama and Llama examples I noticed that weights are always cast to float16, regardless of the format weights are loaded:
- TinyLlama weights are stored in
float32 - Llama weights are stored in
bfloat16
I'd suggest to keep those dtypes in convert.py.
For float32 this should be straight forward. Handling bfloat16 is a bit more challenging.
- np does not support
bfloat16. Sincenp.savezis used, that would need to be changed.
Fortunately, mx hassavezimplemented and it works forbfloat16, so I'd suggest to use that one instead. - converting from torch is currently done with
v = v.to(torch.float16).numpy(). To convert to mx without precision loss, we would need to change this tov = mx.array(v.to(torch.float32).numpy(), dtype=mx.bfloat16)to avoid any intermediate rounding. (Maybe there is a better method without np as intermediate data format.)
So I am proposing to move from np.savez to mx.savez in all examples, where applicable and try to keep the original dtype of the models unless explicit conversion like in quantization is requested.
A quick try out shows that computation in bloat16 does actually generate different tokens (llama-2-13b-chat):
In the beginning the Universe was created.
This has made a great deal of people very angry and been widely regarded as a bad move.
-- Douglas Noble, "The Book of Infinite wisdom"
< I think this is a great example of a joke that is both funny and thought-proving. It takes a familiar concept (the beginning of the Univere) and subverts it in a way that is both surprising and insightful. The puncutation and wording also add to the
---
> I think this is a great example of a joke that is both funny and thought-proving. It takes a well-known concept (the beginning of the Univere) and adds a new and absurd spin to it, creating a humorously absurd situation. At the same time, it also
As expected, changing precision has more impact on tokens generated later.
I am happy to create a PR if there is positive feedback to the proposal.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in convert.py and inspect the model-conversion paths that currently cast weights to float16 and use np.savez. Review the examples mentioned in the issue and determine where mx.savez is applicable, including bfloat16 conversion from torch. Done means preserving the source dtype unless explicit quantization requests conversion and validating the affected examples.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100