Weird Loss with LISA

Hi ,

so giving a background -
I am using Mistral 7b along with HF trainer for finetuning on domain specific data.
Where the task is CausalLM ie next word prediction.
Using datacollatorfor Causal LM for data prep using context size is 1000 tokens per data point and I have 9k total dataset. which includes 5-10% of Wiki data for mixing it with Domain data for avoiding Catastrophic Forgetting.
Test data is a part of train to make it learn on the specific data

I am utilizing the DynamicLayerActivationCallback from LMFlow in my trainer as Training Callbacks.

I tried multiple experiments with -

• lisa_activated_layers- 2 , lisa_interval_steps - 50 epoch 8
• lisa_activated_layers- 2 , lisa_interval_steps - 50 epoch 10
  for both of the runs the loss starts around 8 ad goes around 5-6 but it goes into plateau . and doesnt come below 5 .

I find it little strange, maybe need other experimentation on -

• changing lisa_activated_layers
• changing interval steps (I think this can be important factor too)

Also would like to get the idea, whats the ideal or recommended hyperparams for such type of finetuning with around 10K datapoints.

Thanks in Advance

Start by reproducing the reported Mistral 7B fine-tuning run with Hugging Face Trainer and DynamicLayerActivationCallback, using the listed layer counts, interval, epochs, and dataset setup. Compare the loss behavior across those settings and inspect whether the plateau is expected or indicates a callback or training configuration problem. Done means identifying a reproducible cause and documenting the validated configuration or correction.