Segmentation fault (core dumped) when sdxl-turbo inference with torch 2.2.1+cu118

### Describe the bug

When sdxl-turbo inferencing, I encountered Segmentation fault (core dumped) after loading the model. And this will happen with torch 2.2.1+cu118(xformers0.0.25+cu118), and it will not happen with torch 2.0.1+cu118(xformers0.0.20+cu118). However, I need to run under torch 2.2.1, so could anyone help me to solve this issue? Thanx!

### Reproduction

code:
pipe = AutoPipelineForText2Image.from_pretrained( cache_dir, torch_dtype=torch.float16, variant="fp16")

pipe.to("cuda")

prompt = "A cinematic shot of a baby racoon wearing an intricate italian priest robe."

image = pipe(prompt=prompt, num_inference_steps=1, guidance_scale=0.0).images[0]

### Logs

```shell
Loading pipeline components...: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7/7 [00:01<00:00, 4.47it/s]
0%| | 0/1 [00:00