ml-explore / ml-explore/mlx-examples

Enable the Mixtral-like Moe model without the quantized gate layer

Open
#394 42 comments 4 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
9k
Forks
1.2k
PR merge metrics
No merged PRs in 30d

Description

Currently, the community has started experimenting with building more models using a mix of different local experts. In the current implementation of mlx-lm, we have hardcoded the linear_class_predicate with 8 as an assumption that there will always be 8 local experts. It would be great if we could read the number of local experts from the configuration and make it adjustable to support varying numbers of local experts. This way, users can use mlx-lm to fine-tune the gate for LoRa. For example, here I have to copy and paste code from mlx-lm in order to make lora moe 4x7b work.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the mlx-lm implementation of linear_class_predicate and the model configuration it currently uses, then compare it with the duplicated logic in utils.py lines 57-66 of the linked mlx-moe example. Done means the number of local experts is read from configuration rather than fixed at 8, while supporting LoRA fine-tuning for varying expert counts.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.