Train with custom models SDXL models with `sdxl_train.py`

I want to use dreambooth caption training on this model: [`segmind/Segmind-Vega`](https://huggingface.co/segmind/Segmind-Vega) with I ran into some state_dict errors.
This is my code:
`!accelerate launch --num_cpu_threads_per_process 1 sd-scripts/sdxl_train.py \
--pretrained_model_name_or_path="segmind/Segmind-Vega" \
--no_half_vae \
--dataset_config="style_03_db.toml" \
--output_dir="results/style_03" \
--flip_aug \
--save_model_as="safetensors" \
--max_train_steps=1600 \
--learning_rate=1e-6 \
--optimizer_type="AdamW8bit" \
--xformers \
--mixed_precision="fp16" \
--cache_latents \
--gradient_checkpointing `

And this is the error:
RuntimeError: Error(s) in loading state_dict for SdxlUNet2DConditionModel:
Missing key(s) in state_dict: ...

I see that in the training script right now only SDXL is supported but custom distilled models are not, is this right?

Start with sdxl_train.py and inspect how the pretrained model is loaded and validated against SdxlUNet2DConditionModel. Reproduce the command with segmind/Segmind-Vega, compare its state_dict with the expected SDXL structure, and define done as successful loading and training of the custom model without missing-key errors.

Written by the indexing model from the issue text.