Fusing adapters with llama3 cause bad performances

Hello,

I'm using the following script to fine tune the llama3 model with a custom dataset of questions & responses using the {'prompt: "", completion:""} format defined here.

#!/usr/bin/env bash

DATA_PATH=data
ADAPTERS_PATH=adapters
MODEL_NAME=meta-llama/Meta-Llama-3-8B
MODEL_PATH=models/mlx
LORA_CONFIG_PATH=lora_config.yaml

NAME="my-assistant"
ITERATIONS=1000

# Parse options
while [[ "$#" -gt 0 ]]; do
    case $1 in
        --name)
            if [[ -z "$2" ]] || [[ "$2" == --* ]]; then
                echo "Error: Model name cannot be empty"
                echo "Usage: fine-tune.sh --iter <number of iterations> --name <output model name>"
                exit 1
            fi
            NAME="$2"
            shift
            ;;
        --iter)
            if [[ "$2" -lt 100 || "$2" -gt 10000 ]]; then
                echo "Error: Iteration value must be an integer between 100 and 10000"
                exit 1
            fi
            ITERATIONS="$2"
            shift
            ;;
        *)
          echo "Unknown option: $1."
          echo "Usage: fine-tune.sh --iter <number of iterations> --name <output model name>"
          exit 1
          ;;
    esac
    shift
done

echo "Fine-tuning with $ITERATIONS iterations"
echo "Output model name: $NAME"
FINE_TUNED_MODEL_PATH=models/$NAME
GGUF_MODEL_PATH=models/$NAME.gguf

set -ex

# Install llama.cpp if needed
if [ ! -d "llama.cpp" ]; then
    git clone git@github.com:ggerganov/llama.cpp.git;
    cd llama.cpp;
    make -j 8;
    cd ..;
else
    echo "Directory llama.cpp found, skipping download and build"
fi

# Download & quantize the HuggingFace model to reduce weights memory footprint
if [ ! -d "$MODEL_PATH" ]; then
    echo "No models found, initiating quantization...";
    python -m mlx_lm.convert \
        --hf-path "$MODEL_NAME" \
        --mlx-path "$MODEL_PATH" \
        -q;
else
  echo "Model found in $MODEL_PATH, skipping quantization"
fi

# Launch fine-tuning
python -m mlx_lm.lora \
    --data "$DATA_PATH" \
    --model "$MODEL_PATH" \
    --train \
    --iters "$ITERATIONS" \
    --config "$LORA_CONFIG_PATH"

# Merge the model and fine-tuned adapter
python -m mlx_lm.fuse \
    --model "$MODEL_PATH" \
    --adapter-path "$ADAPTERS_PATH" \
    --save-path "$FINE_TUNED_MODEL_PATH" \
    --de-quantize

# Generate GGUF file to use the new model with Ollama
python llama.cpp/convert-hf-to-gguf.py "$FINE_TUNED_MODEL_PATH" \
    --outfile "$GGUF_MODEL_PATH" \
    --outtype q8_0 \

ollama create "$NAME" -f Modelfile

I trained the model over 1000 iterations with the following config parameters:

# Number of validation batches, -1 uses the entire validation set.
val_batches: -1

# Adam learning rate.
learning_rate: 1e-5

# Number of training steps between loss reporting.
steps_per_report: 10

# Number of training steps between validations.
steps_per_eval: 50

# Save the model every N iterations.
save_every: 100

# Evaluate on the test set after training
test: true

# Number of test set batches, -1 uses the entire test set.
test_batches: 100

lora_parameters:
  # The layer keys to apply LoRA to.
  # These will be applied for the last lora_layers
  rank: 8
  scale: 20.0
  dropout: 0.0

The Modelfile s below:

FROM ./models/my-assistant.gguf

TEMPLATE """{{ if .System }}<|im_start|>system
{{ .System }}<|im_end|>
{{ end }}{{ if .Prompt }}<|im_start|>user
{{ .Prompt }}<|im_end|>
{{ end }}<|im_start|>assistant
"""

PARAMETER stop "<|im_end|>"
PARAMETER stop "<|im_start|>"
PARAMETER temperature 0

During the training I get the following message but I don't know what I am supposed to do with mlx_lm options:

No chat template is set for this tokenizer, falling back to a default class-level template. This is very error-prone, because models are often trained with templates different from the class default! Default chat templates are a legacy feature and will be removed in Transformers v4.43, at which point any code depending on them will stop working. We recommend setting a valid chat template before then to ensure that this model continues working without issues.

The final loss on training set (1300 samples) is around 0.16 and over validation set is 0.26. When running the generate command :

python -m mlx_lm.generate --model models/mlx --adapter-path adapters --prompt "<question>" 

I get very good results and nothing to complain about.

However when I use the fused model generated by my script using this documentation, the performances are super bad and answer are using random values when asking for specific amounts the model was trained for. The performances are as bad as if I didn't do any fine tuning.

Did I do something wrong or is the fusing process supposed to be that bad ? I there a way to export the model used by the mlx_lm.generate command to GGUF instead of relying on mlx_lm.fuse ?

Thank you.

Reproduce the discrepancy using the shown mlx_lm.generate command with adapters and the mlx_lm.fuse command, then inspect the fused model before conversion with llama.cpp/convert-hf-to-gguf.py. Compare generation outputs and model settings, including the chat template warning. Done means the fused and adapter-backed models produce equivalent results for the reported prompts.