(QLoRA) DPO without previous SFT

Because of the following LLM-Leaderboard measurements, I want to perform QLoRA DPO without previous QLoRA SFT:
```
alignment-handbook/zephyr-7b-dpo-qlora: +Average: 63.51; +ARC 63.65; +HSwag 85.35; -+MMLU 63.82; ++TQA: 47.14; (+)Win 79.01; +GSM8K 42.08;

alignment-handbook/zephyr-7b-sft-qlora: -Average: 59; (+)ARC 60.07; (-)HSwag 82.36; -MMLU 61.65; -TQA: 38.88; -Win 76.8; -GSM8K 34.27;

mistralai/Mistral-7B-v0.1: Average: 60.97; ARC 59.98; HSwag 83.31; MMLU 64.16; TQA: 42.15; Win 78.37; GSM8K 37.83;
```
As you can see, there is catastrophic forgetting in `zephyr-7b-sft-qlora` in almost all tasks, especially in MMLU, TruthfulQA, and GSM8K. Thus I wonder why do SFT at all?

In more detail
============

Q1: Why is there so much catastrophic forgetting in `zephyr-7b-sft-qlora` ? Due to the following improvements by DPO, the dataset seems to be apt.

Q2: Why is SFT performed before DPO at all? Is it some prerequisite, like SFT training the model to follow instructions at all, before DPO aligning the responses to instructions with human preferences?

Q3: I tried the following for DPO without previous SFT:
Modify `recipes/zephyr-7b-beta/dpo/config_qlora.yaml` by using `model_name_or_path: mistralai/Mistral-7B-v0.1` and then calling `scripts/run_dpo.py` on it:
```
echo -e "2,3c2\n< model_name_or_path: mistralai/Mistral-7B-v0.1\n< model_revision: main\n---\n> model_name_or_path: alignment-handbook/zephyr-7b-sft-qlora\n36c35\n< gradient_accumulation_steps: 8\n---\n> gradient_accumulation_steps: 2\n40c39\n< hub_model_id: zephyr-7b-dpo-qlora-no-sft\n---\n> hub_model_id: zephyr-7b-dpo-qlora\n49,51c48,50\n< output_dir: data/zephyr-7b-dpo-qlora-no-sft # It is handy to append `hub_model_revision` to keep track of your local experiments\n< per_device_train_batch_size: 1\n< per_device_eval_batch_size: 2\n---\n> output_dir: data/zephyr-7b-dpo-qlora # It is handy to append `hub_model_revision` to keep track of your local experiments\n> per_device_train_batch_size: 4\n> per_device_eval_batch_size: 8\n53,55d51\n< report_to:\n< - tensorboard\n< - wandb" | patch recipes/zephyr-7b-beta/dpo/config_qlora.yaml
ACCELERATE_LOG_LEVEL=info accelerate launch --config_file recipes/accelerate_configs/multi_gpu.yaml --num_processes=1 scripts/run_dpo.py recipes/zephyr-7b-beta/dpo/config_qlora.yaml
```
However, I get the error described at https://github.com/huggingface/alignment-handbook/issues/93. The solution there inspired me to do the following (so I don't have to go into the cache to replace tokenizer configs): Add in line 77 of `src/alignment/data.py`
```
tokenizer.chat_template = "{% for message in messages %}\n{% if message['role'] == 'user' %}\n{{ '<|user|>\n' + message['conten\
t'] + eos_token }}\n{% elif message['role'] == 'system' %}\n{{ '<|system|>\n' + message['content'] + eos_token }}\n{% elif message['role'] \
== 'assistant' %}\n{{ '<|assistant|>\n' + message['content'] + eos_token }}\n{% endif %}\n{% if loop.last and add_generation_prompt %}\n{{\
'<|assistant|>' }}\n{% endif %}\n{% endfor %}"
```
But Mistral's `default_chat_template` already allows system messages, so the problem seems to be that the dialogs in the dataset really do not alternate between user and assistant messages. Right? What is the reason for this?

Mistrals `default_chat_template` causing the error message:
```
{% if messages[0]['role'] == 'system' %}{% set loop_messages = messages[1:] %}{% set system_message = messages[0]['content'] %}{% elif false == true and not '<>' in messages[0]['content'] %}{% set loop_messages = messages %}{% set system_message = 'You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe. Your answers should not include any harmful, unethical,
racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature.\n\nIf a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don\'t know
the answer to a question, please don\'t share false information.' %}{% else %}{% set loop_messages = messages %}{% set system_message = false %}{% endif %}{% for message in loop_messages %}{% if (message['role'] == 'user') != (loop.index0 % 2 == 0) %}{{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}{% endif %}{% if loop.index0 == 0 and system_message != false %}{% set conte
nt = '<>\n' + system_message + '\n<>\n\n' + message['content'] %}{% else %}{% set content = message['content'] %}{% endif %}{% if message['role'] == 'user' %}{{ bos_token + '[INST] ' + content.strip() + ' [/INST]' }}{% elif message['role'] == 'system' %}{{ '<>\n' + content.strip() + '\n<>\n\n' }}{% elif message['role'] == 'assistant' %}{{ ' ' + content.strip() + ' ' + eos_token }}{% endif %}{%
endfor %}
```