cuDNN error: CUDNN_STATUS_NOT_INITIALIZED

I run the training like below, but throught out an Erro: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED .
torchrun --num_processes 1 train_network.py \
--pretrained_model_name_or_path=/aigc2/liutl/model/dreamshaper_631BakedVae.safetensors \
--dataset_config=./timeprinces.toml \
--tokenizer_cache_dir=/aigc2/liutl/model/clip-vit-large-patch14 \
--output_dir=./timeprince_model/ \
--output_name=yeluolishixi \
--save_model_as=safetensors \
--prior_loss_weight=1.0 \
--max_train_steps=400 \
--learning_rate=1e-4 \
--xformers \
--mixed_precision="fp16" \
--cache_latents \
--gradient_checkpointing \
--save_every_n_epochs=1 \
--network_module=networks.lora

(sd-script) root@ecp-lite-gpu1080-2:/aigc2/liutl/sd-scripts# cat timeprinces.toml
[general]
enable_bucket = true # 是否使用Aspect Ratio Bucketing

[[datasets]]
resolution = 512 # 学习分辨率
batch_size = 4 # 批量大小

[[datasets.subsets]]
image_dir = '/aigc2/liutl/sd/train_data/6_timeprincess_girl' # 指定包含训练图像的文件夹
class_tokens = 'timeprince 1girl' # 指定标识符类
num_repeats = 6 # 训练图像的迭代次数

# 以下仅在使用正则化图像时进行描述。不使用则删除
[[datasets.subsets]]
is_reg = false
image_dir = '' # 指定包含正则化图像的文件夹
class_tokens = 'girl' # 指定类别
num_repeats = 1 # 正则化图像的迭代次数，基本上1就可以了

import network module: networks.lora
[Dataset 0]
caching latents.
0%| | 0/20 [00:00
train(args)
File "/aigc2/liutl/sd-scripts/train_network.py", line 203, in train
train_dataset_group.cache_latents(vae, args.vae_batch_size, args.cache_latents_to_disk, accelerator.is_main_process)
File "/aigc2/liutl/sd-scripts/library/train_util.py", line 1422, in cache_latents
dataset.cache_latents(vae, vae_batch_size, cache_to_disk, is_main_process)
File "/aigc2/liutl/sd-scripts/library/train_util.py", line 814, in cache_latents
latents = vae.encode(img_tensors).latent_dist.sample().to("cpu")
File "/root/anaconda3/envs/sd-script/lib/python3.10/site-packages/diffusers/models/vae.py", line 566, in encode
h = self.encoder(x)
File "/root/anaconda3/envs/sd-script/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/root/anaconda3/envs/sd-script/lib/python3.10/site-packages/diffusers/models/vae.py", line 130, in forward
sample = self.conv_in(sample)
File "/root/anaconda3/envs/sd-script/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/root/anaconda3/envs/sd-script/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 463, in forward
return self._conv_forward(input, self.weight, self.bias)
File "/root/anaconda3/envs/sd-script/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 459, in _conv_forward
return F.conv2d(input, weight, bias, self.stride,
RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 1085555) of binary: /root/anaconda3/envs/sd-script/bin/python
Traceback (most recent call last):
File "/root/anaconda3/envs/sd-script/bin/torchrun", line 8, in
sys.exit(main())
File "/root/anaconda3/envs/sd-script/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
return f(*args, **kwargs)
File "/root/anaconda3/envs/sd-script/lib/python3.10/site-packages/torch/distributed/run.py", line 794, in main
run(args)
File "/root/anaconda3/envs/sd-script/lib/python3.10/site-packages/torch/distributed/run.py", line 785, in run
elastic_launch(
File "/root/anaconda3/envs/sd-script/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
return launch_agent(self._config, self._entrypoint, list(args))
File "/root/anaconda3/envs/sd-script/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================

Reproduce the command shown in the issue and start at train_network.py:203, then follow library/train_util.py:1422 and library/train_util.py:814 into the failing VAE encode call. Compare the environment and GPU conditions around the PyTorch conv2d failure; done means identifying a reproducible cause and a verified resolution, since the issue does not specify a code change.

Written by the indexing model from the issue text.