nan in diffusers case

### 🐛 Describe the bug

tests/pipelines/dreamlite/test_pipeline_dreamlite.py::TestDreamLitePipelineIntegration::test_dreamlite_t2i_real_checkpoint
tests/pipelines/dreamlite/test_pipeline_dreamlite_mobile.py::TestDreamLiteMobilePipelineIntegration::test_mobile_t2i_real_checkpoint

git clone https://github.com/sywangyi/diffusers.git
cd diffusers
git checkout -b extend_test_to_xpu origin/extend_test_to_xpu

uv pip install -e ".[test]"

export RUN_SLOW=true
export RUN_NIGHTLY=true

pytest tests/pipelines/dreamlite/test_pipeline_dreamlite.py::TestDreamLitePipelineIntegration::test_dreamlite_t2i_real_checkpoint

pytest tests/pipelines/dreamlite/test_pipeline_dreamlite_mobile.py::TestDreamLiteMobilePipelineIntegration::test_mobile_t2i_real_checkpoint

two cases fail because of nan

if you export DIFFUSERS_ATTN_BACKEND=_native_math change to math attn backend, these two case will pass.

### Versions

Collecting environment information...
PyTorch version: 2.15.0.dev20260829+xpu
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: Ubuntu 24.04.2 LTS (x86_64)
GCC version: (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.39

Python version: 3.12.3 (main, Jul 15 2026, 23:46:41) [GCC 13.3.0] (64-bit runtime)
Python platform: Linux-7.0.0-30-generic-x86_64-with-glibc2.39
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
Is XPU available: True
XPU used to build PyTorch: 20260100
Intel GPU driver version:
* intel-opencl-icd: 26.27.39122.14-1~24.04~ppa1
* libze1: 1.32.0-1~24.04~ppa1
Intel GPU models onboard:
* Intel(R) Arc(TM) Pro B70 Graphics
* Intel(R) Arc(TM) Pro B70 Graphics
* Intel(R) Arc(TM) Pro B70 Graphics
* Intel(R) Arc(TM) Pro B70 Graphics
Intel GPU models detected:
* [0] _XpuDeviceProperties(name='Intel(R) Arc(TM) Pro B70 Graphics', platform_name='Intel(R) oneAPI Unified Runtime over Level-Zero V2', type='gpu', device_id=0xE223, uuid=868023e2-0000-0000-0d00-000000000000, driver_version='1.15.39122+14', total_memory=32656MB, local_mem_size=128KB, last_level_cache_size=24576KB, max_compute_units=256, memory_clock_rate=2800MHz, memory_bus_width=64-bit, gpu_eu_count=256, gpu_subslice_count=32, max_work_group_size=1024, max_num_sub_groups=64, sub_group_sizes=[16 32], has_fp16=1, has_fp64=1, has_atomic64=1, is_integrated_gpu=0)
* [1] _XpuDeviceProperties(name='Intel(R) Arc(TM) Pro B70 Graphics', platform_name='Intel(R) oneAPI Unified Runtime over Level-Zero V2', type='gpu', device_id=0xE223, uuid=868023e2-0000-0000-4800-000000000000, driver_version='1.15.39122+14', total_memory=32656MB, local_mem_size=128KB, last_level_cache_size=24576KB, max_compute_units=256, memory_clock_rate=2800MHz, memory_bus_width=64-bit, gpu_eu_count=256, gpu_subslice_count=32, max_work_group_size=1024, max_num_sub_groups=64, sub_group_sizes=[16 32], has_fp16=1, has_fp64=1, has_atomic64=1, is_integrated_gpu=0)
* [2] _XpuDeviceProperties(name='Intel(R) Arc(TM) Pro B70 Graphics', platform_name='Intel(R) oneAPI Unified Runtime over Level-Zero V2', type='gpu', device_id=0xE223, uuid=868023e2-0000-0000-6200-000000000000, driver_version='1.15.39122+14', total_memory=32656MB, local_mem_size=128KB, last_level_cache_size=24576KB, max_compute_units=256, memory_clock_rate=2800MHz, memory_bus_width=64-bit, gpu_eu_count=256, gpu_subslice_count=32, max_work_group_size=1024, max_num_sub_groups=64, sub_group_sizes=[16 32], has_fp16=1, has_fp64=1, has_atomic64=1, is_integrated_gpu=0)
* [3] _XpuDeviceProperties(name='Intel(R) Arc(TM) Pro B70 Graphics', platform_name='Intel(R) oneAPI Unified Runtime over Level-Zero V2', type='gpu', device_id=0xE223, uuid=868023e2-0000-0000-b000-000000000000, driver_version='1.15.39122+14', total_memory=32656MB, local_mem_size=128KB, last_level_cache_size=24576KB, max_compute_units=256, memory_clock_rate=2800MHz, memory_bus_width=64-bit, gpu_eu_count=256, gpu_subslice_count=32, max_work_group_size=1024, max_num_sub_groups=64, sub_group_sizes=[16 32], has_fp16=1, has_fp64=1, has_atomic64=1, is_integrated_gpu=0)
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: False
Caching allocator config: N/A

Start with tests/pipelines/dreamlite/test_pipeline_dreamlite.py::TestDreamLitePipelineIntegration::test_dreamlite_t2i_real_checkpoint and tests/pipelines/dreamlite/test_pipeline_dreamlite_mobile.py::TestDreamLiteMobilePipelineIntegration::test_mobile_t2i_real_checkpoint. Reproduce the failures on XPU with RUN_SLOW=true and RUN_NIGHTLY=true, compare the _native_math and math attention backends, and consider the work done when both tests pass without NaN under the default backend.

Written by the indexing model from the issue text.