ml-explore / ml-explore/mlx-examples

Keep `dtype` of Models

Open
#179 7 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
9k
Forks
1.2k
PR merge metrics
No merged PRs in 30d

Description

While playing around with the TinyLlama and Llama examples I noticed that weights are always cast to float16, regardless of the format weights are loaded:

  • TinyLlama weights are stored in float32
  • Llama weights are stored in bfloat16

I'd suggest to keep those dtypes in convert.py.

For float32 this should be straight forward. Handling bfloat16 is a bit more challenging.

  1. np does not support bfloat16. Since np.savez is used, that would need to be changed.
    Fortunately, mx has savez implemented and it works for bfloat16, so I'd suggest to use that one instead.
  2. converting from torch is currently done with v = v.to(torch.float16).numpy(). To convert to mx without precision loss, we would need to change this to v = mx.array(v.to(torch.float32).numpy(), dtype=mx.bfloat16) to avoid any intermediate rounding. (Maybe there is a better method without np as intermediate data format.)

So I am proposing to move from np.savez to mx.savez in all examples, where applicable and try to keep the original dtype of the models unless explicit conversion like in quantization is requested.

A quick try out shows that computation in bloat16 does actually generate different tokens (llama-2-13b-chat):

In the beginning the Universe was created.
This has made a great deal of people very angry and been widely regarded as a bad move.

 -- Douglas Noble, "The Book of Infinite wisdom"

< I think this is a great example of a joke that is both funny and thought-proving. It takes a familiar concept (the beginning of the Univere) and subverts it in a way that is both surprising and insightful. The puncutation and wording also add to the
---
> I think this is a great example of a joke that is both funny and thought-proving. It takes a well-known concept (the beginning of the Univere) and adds a new and absurd spin to it, creating a humorously absurd situation. At the same time, it also

As expected, changing precision has more impact on tokens generated later.

I am happy to create a PR if there is positive feedback to the proposal.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in convert.py and inspect the model-conversion paths that currently cast weights to float16 and use np.savez. Review the examples mentioned in the issue and determine where mx.savez is applicable, including bfloat16 conversion from torch. Done means preserving the source dtype unless explicit quantization requests conversion and validating the affected examples.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.