Ability to convert a lora_fused_model to gguf format for use in LMStudio and others

Recently I got a flow working where I would train a model with mlx (this is new for me) and then move over to llama.cpp to do the conversion to gguf in order to run it on LMStudio locally. However with Mixtral in particular the conversion is failing, and I am not sure if it's something wrong with training or llama.cpp, or something new to do with the MoE architecture.

Here's how to reproduce (I'm on an M3 Macbook Pro). In the mlx_examples/lora folder:

python convert.py --hf-path mistralai/Mixtral-8x7B-Instruct-v0.1 -q

Which downloads the quantized model locally to /mlx_model (I have 64gb memory but I found it would crash if I don't use the quantized model).

Then I can run the following training script:

python lora.py \
 --train \
 --model mlx_model \
 --data ./data \
 --batch-size 2 \
 --lora-layers 8 \
 --iters 1000

In /data I have train.jsonl and valid.jsonl like in the examples, and I tested it worked fine on Mistral-7B-v0.1 so I know this part is fine. I'm not allowed to share the data but you could probably just run this with the existing example data to replicate.

When I run fuse with the dequantize parameter (I've tried with and without) I get the /lora_fused_model folder

python fuse.py --model mlx_model \
 --adapter-file ./adapters.npz \
 --de-quantize

Normally what I would do now is move that over to llama.cpp and use convert.py to get the f16 version:

python convert.py models/lora_fused_model --outfile models/lora_fused_model-fp16.gguf --outtype f16

Then I would quantize further to get the Q4_K_M version that I normally use in LM Studio.

./quantize ./models/lora_fused_model-fp16.gguf ./models/lora_fused_model.Q4_K_M.gguf Q4_K_M

However, when I did that I got the error Exception: Unexpected tensor name: model.layers.0.block_sparse_moe.experts.0.w1.weight.

I upgraded llama.cpp and then ran it again and got this error: "GGUFWriter" object has no attribute "add_expert_count"

I don't know if this is an issue with llama.cpp, or if I did something wrong in the training or conversion process which meant llama.cpp can't convert it? Perhaps there's a way to convert to gguf in mlx that I missed? I have seen fine-tunes of Mixtral-8x7b out there so I presume it's possible, but maybe the MoE format isn't supported yet or needs me to run a different script?