Strange errors occurred during inference

Loading pretrained ckpt from ./ckpts/seedvr2_ema_7b.pth
Loading info:
DiT trainable parameters: 8,239,607,872
2025-06-17 12:50:17,425 [MainThread][common.decorators][INFO] Completed configure_dit_model in 135.817 seconds.
2025-06-17 12:50:17,425 [MainThread][common.decorators][INFO] Entering configure_vae_model
/root/anaconda3/envs/seedvr/lib/python3.10/site-packages/diffusers/models/transformers/transformer_2d.py:34: FutureWarning: `Transformer2DModelOutput` is depr
ecated and will be removed in version 1.0.0. Importing `Transformer2DModelOutput` from `diffusers.models.transformer_2d` is deprecated and this will be remove
d in a future version. Please use `from diffusers.models.modeling_outputs import Transformer2DModelOutput`, instead.
deprecate("Transformer2DModelOutput", "1.0.0", deprecation_message)
2025-06-17 12:50:17.957417: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to float
ing-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2025-06-17 12:50:17.990382: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in perf
ormance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-06-17 12:50:19.875329: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
slicing_up_num: 0
2025-06-17 12:50:29,849 [MainThread][common.decorators][INFO] Completed configure_vae_model in 12.424 seconds.
Total prompts to be generated: 1
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 97.40it/s]
0it [00:00, ?it/s]/root/anaconda3/envs/seedvr/lib/python3.10/site-packages/torchvision/io/video.py:161: UserWarning: The pts_unit 'pts' gives wrong results. P
lease use pts_unit 'sec'.
warnings.warn("The pts_unit 'pts' gives wrong results. Please use pts_unit 'sec'.")
Read video size: torch.Size([101, 3, 960, 540])
Encoding videos: [torch.Size([3, 101, 1280, 720])]
/root/anaconda3/envs/seedvr/lib/python3.10/site-packages/torch/nn/modules/conv.py:605: UserWarning: Plan failed with a cudnnException: CUDNN_BACKEND_EXECUTION
_PLAN_DESCRIPTOR: cudnnFinalize Descriptor Failed cudnn_status: CUDNN_STATUS_NOT_SUPPORTED (Triggered internally at ../aten/src/ATen/native/cudnn/Conv_v8.cpp:
919.)
return F.conv3d(
Generating with noise shape: torch.Size([26, 160, 90, 16]).

Timestep shifting from 0.0 to tensor([0.], device='cuda:0').
0617 12:51:11.199000 140509054621504 torch/distributed/elastic/multiprocessing/api.py:826] failed (exitcode: -8) local_rank: 0 (pid: 436) of binary: /root/ana
conda3/envs/seedvr/bin/python3.10
Traceback (most recent call last):
File "/root/anaconda3/envs/seedvr/bin/torchrun", line 8, in
sys.exit(main())
File "/root/anaconda3/envs/seedvr/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper
return f(*args, **kwargs)
File "/root/anaconda3/envs/seedvr/lib/python3.10/site-packages/torch/distributed/run.py", line 879, in main
run(args)
File "/root/anaconda3/envs/seedvr/lib/python3.10/site-packages/torch/distributed/run.py", line 870, in run
elastic_launch(
File "/root/anaconda3/envs/seedvr/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 132, in __call__
return launch_agent(self._config, self._entrypoint, list(args))
File "/root/anaconda3/envs/seedvr/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 263, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:

projects/inference_seedvr2_7b.py FAILED

Failures:

Root Cause (first observed failure):
[0]:
time : 2025-06-17_12:51:11
host : VM-63-196-tencentos
rank : 0 (local_rank: 0)
exitcode : -8 (pid: 436)
error_file:
traceback : Signal 8 (SIGFPE) received by PID 436