Gradient Clipping Doesn't Work in Finetuning

This one's frustrating because the config files SHOW a max_norm parameter, but it doesn't actually do anything during finetuning.

# litgpt/finetune/lora.py:560
unsupported = [(train, ["max_tokens", "max_norm", "tie_embeddings", "lr_warmup_fraction"])]

Meanwhile in pretraining:

# litgpt/pretrain.py:360
fabric.clip_gradients(model, optimizer, max_norm=train.max_norm)

Why this is a problem:

Gradient clipping is pretty important for stable training, especially with:

• Small datasets (where you're more likely to hit bad batches)
• Long sequences
• QLoRA (where quantization noise can cause gradient spikes)
• Any time you're near the edge of numerical stability

The config files in config_hub/finetune/ all have max_norm: set to empty/null. Users will assume this means "no clipping" when it actually means "this parameter is ignored."

The fix:

Just implement it. The code's already there in pretrain.py - copy it over. Make sure it works with:

• Regular LoRA
• QLoRA
• Full finetuning
• Adapter training

Or if you really don't want to support it, remove it from the configs and document why.

Start by comparing litgpt/finetune/lora.py around line 560 with the clipping call in litgpt/pretrain.py around line 360. Trace how max_norm is loaded from the finetuning configs, then verify behavior for regular LoRA, QLoRA, full finetuning, and adapter training. Done means max_norm controls clipping in these modes, or is removed from the configs with its unsupported status documented.