Missing f8 dtypes
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 28.5k
- Forks
- 2.3k
- Avg merge
- 3d 8h
- Merged PRs (30d)
- 62
Description
Hi, the unified memory of Apple silicon devices is compelling for AI training, and often enables these to have significantly more memory for parameters and gradients than best consumer or even data center grade discrete GPUs.
However, inspecting the data types list, the smallest float dtype I saw in mlx today was 16 bits (f16 or bf16)
Adding 8 bit floats to mlx would effectively double the maximum possible model size.
Would this be possible with software or does it need to be a hardware update?
If it is possible with software, how could we make it happen?
Some options for sensible default f8 e/m split for an 8-bit dtype could be:
e5m2
e4m3
e3m4
Then the question becomes how we would rank these and decide which one is best for the most likely use cases?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing MLX's existing data type list and how its f16 and bf16 types are represented. Determine whether e5m2, e4m3, or e3m4 is supported by software and hardware, and compare their likely training use cases. Done means a selected f8 design and a clear implementation path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100