[Bug] Input type (c10::BFloat16) and bias type (float) should be the same

```python
epoch 1/200
/home/ubuntu/tools/sd-scripts/.venv/lib/python3.10/site-packages/torch/utils/checkpoint.py:31: UserWarning: None of the inputs have requires_grad=True. Gradients will be None
warnings.warn("None of the inputs have requires_grad=True. Gradients will be None")
Traceback (most recent call last):
File "/home/ubuntu/tools/sd-scripts/fine_tune.py", line 437, in
train(args)
File "/home/ubuntu/tools/sd-scripts/fine_tune.py", line 314, in train
noise_pred = unet(noisy_latents, timesteps, encoder_hidden_states).sample
File "/home/ubuntu/tools/sd-scripts/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/ubuntu/tools/sd-scripts/.venv/lib/python3.10/site-packages/accelerate/utils/operations.py", line 490, in __call__
return convert_to_fp32(self.model_forward(*args, **kwargs))
File "/home/ubuntu/tools/sd-scripts/.venv/lib/python3.10/site-packages/torch/amp/autocast_mode.py", line 14, in decorate_autocast
return func(*args, **kwargs)
File "/home/ubuntu/tools/sd-scripts/.venv/lib/python3.10/site-packages/diffusers/models/unet_2d_condition.py", line 375, in forward
sample = self.conv_in(sample)
File "/home/ubuntu/tools/sd-scripts/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/ubuntu/tools/sd-scripts/.venv/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 463, in forward
return self._conv_forward(input, self.weight, self.bias)
File "/home/ubuntu/tools/sd-scripts/.venv/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 459, in _conv_forward
return F.conv2d(input, weight, bias, self.stride,
RuntimeError: Input type (c10::BFloat16) and bias type (float) should be the same
steps: 0%| | 0/146000 [17:24
sys.exit(main())
File "/home/ubuntu/tools/sd-scripts/.venv/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 45, in main
args.func(args)
File "/home/ubuntu/tools/sd-scripts/.venv/lib/python3.10/site-packages/accelerate/commands/launch.py", line 1104, in launch_command
simple_launcher(args)
File "/home/ubuntu/tools/sd-scripts/.venv/lib/python3.10/site-packages/accelerate/commands/launch.py", line 567, in simple_launcher
raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['/home/ubuntu/tools/sd-scripts/.venv/bin/python3.10', 'fine_tune.py', '--config_file', '/home/ubuntu/unsplash/training/toml_files/config.toml']' returned non-zero exit status 1.
```

Anyone can help with this?

I am using bf16 as my precision argument. If needed I can provide my config.toml file as well

Start with fine_tune.py at lines 314 and 437, then review the supplied traceback and the user's config.toml with the precision and environment settings. Reproduce the failure using the reported bf16 configuration and inspect the model call at the conv_in failure. Done means training starts without the input, weight, and bias dtype mismatch.

Written by the indexing model from the issue text.