Bad Flux lora training result

Hi@kohya-ss , thank you for your detailed and excellent work on FLUX finetuning and lora training!! I got bad results when I ran the sample lora_training script with network_dim=32, input 50 high-quality 1024 * 1024 images, max_train_epochs=50 ( 2500steps in total )

python==3.10.15
torch==2.4.0
torchmetrics ==1.6.0
torchvision==0.19.0
transformers==4.44.0
accelerate==0.33.0
xformers== 0.0.23.post1
diffusers==0.25.0

`CUDA_VISIBLE_DEVICES=0 accelerate launch --num_processes 1 --main_process_port 23333 \
flux_train_network.py \
--pretrained_model_name_or_path /black-forest-labs/FLUX.1-schnell/flux1-schnell.safetensors \
--clip_l /SD3/text_encoders/clip_l.safetensors \
--t5xxl /SD3/text_encoders/t5xxl_fp16.safetensors \
--ae /black-forest-labs/FLUX.1-schnell/ae.safetensors \
--cache_latents_to_disk \
--save_model_as safetensors \
--sdpa \
--persistent_data_loader_workers \
--max_data_loader_n_workers 2 \
--seed 42 \
--gradient_checkpointing \
--mixed_precision bf16 \
--save_precision bf16 \
--network_module networks.lora_flux \
--network_dim 32 \
--network_train_unet_only \
--optimizer_type adamw8bit \
--learning_rate 1e-4 \
--cache_text_encoder_outputs \
--cache_text_encoder_outputs_to_disk \
--highvram \
--max_train_epochs 50 \
--save_every_n_epochs 1 \
--dataset_config flux_image_50.toml \
--output_dir /flux_unet/log/lora \
--output_name flux-lora-name \
--timestep_sampling shift \
--discrete_flow_shift 3.1582 \
--model_prediction_type raw \
--guidance_scale 1.0 \`

I got bad inference result when running
`python3 flux_minimal_inference.py --ckp black-forest-labs/FLUX.1-schnell/flux1-schnell.safetensors --clip_l /SD3/text_encoders/clip_l.safetensors --t5xxl /SD3/text_encoders/t5xxl_fp16.safetensors --ae /black-forest-labs/FLUX.1-schnell/ae.safetensors --dtype bf16 --prompt "A small cactus with a happy face in the Sahara desert." --out /flux_unet/log/lora --seed 42 --flux_dtype fp8 --offload --lora /flux_unet/log/lora/flux-lora-name.safetensors;1.0`

The comparison between the original FLUX output ( upper ) and the lora-added output (lower) is

whereas my training images are very good (like this)

Can you give me some hints about it ? thank you so much !!!!!

Reproduce the report with flux_train_network.py, the flux_image_50.toml dataset configuration, and flux_minimal_inference.py using the commands and versions provided. Compare the original and LoRA-added outputs, then identify the training or inference setting responsible and document a verified correction.

Written by the indexing model from the issue text.