[BUG]RuntimeError: Distributed package doesn't have NCCL built in. Training on huawei NPU

When I run "deepspeed start_train.py" with huawei NPU, a runtimeerror accurs, the error shows that Distributed package doesn't have NCCL built in. I would like to know whether I'm using DeepSpeed incorrectly, or if it's due to an unsupported environment.

The log is as following:

[2025-05-26 13:54:12,447] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to npu (auto detect)
[2025-05-26 13:54:12,573] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to npu (auto detect)
[2025-05-26 13:54:12,665] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to npu (auto detect)
[2025-05-26 13:54:12,855] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to npu (auto detect)
[2025-05-26 13:54:12,855] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to npu (auto detect)
[2025-05-26 13:54:12,892] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to npu (auto detect)
[2025-05-26 13:54:12,929] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to npu (auto detect)
[2025-05-26 13:54:13,338] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to npu (auto detect)
[2025-05-26 13:54:17,144] [INFO] [comm.py:637:init_distributed] cdb=None
[2025-05-26 13:54:17,412] [INFO] [comm.py:637:init_distributed] cdb=None
[2025-05-26 13:54:17,413] [INFO] [comm.py:668:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl
[2025-05-26 13:54:17,475] [INFO] [comm.py:637:init_distributed] cdb=None
Traceback (most recent call last):
File "/home/gq/crossformer_qwen7b_llava/start_train.py", line 62, in
start_train()
File "/home/gq/crossformer_qwen7b_llava/start_train.py", line 56, in start_train
train()
File "/home/gq/crossformer_qwen7b_llava/llava_qwen7b_module/llava/train/train.py", line 981, in train
model_args, data_args, training_args = parser.parse_args_into_dataclasses()
File "/usr/local/python3.10.13/lib/python3.10/site-packages/transformers/hf_argparser.py", line 338, in parse_args_into_dataclasses
obj = dtype(**inputs)
File "", line 136, in init
File "/usr/local/python3.10.13/lib/python3.10/site-packages/transformers/training_args.py", line 1483, in post_init
and (self.device.type != "cuda")
File "/usr/local/python3.10.13/lib/python3.10/site-packages/transformers/training_args.py", line 1921, in device
return self._setup_devices
File "/usr/local/python3.10.13/lib/python3.10/site-packages/transformers/utils/generic.py", line 54, in get
cached = self.fget(obj)
File "/usr/local/python3.10.13/lib/python3.10/site-packages/transformers/training_args.py", line 1853, in _setup_devices
self.distributed_state = PartialState(timeout=timedelta(seconds=self.ddp_timeout))
File "/usr/local/python3.10.13/lib/python3.10/site-packages/accelerate/state.py", line 170, in init
dist.init_distributed(dist_backend=self.backend, auto_mpi_discovery=False, **kwargs)
File "/usr/local/python3.10.13/lib/python3.10/site-packages/deepspeed/comm/comm.py", line 670, in init_distributed
cdb = TorchBackend(dist_backend, timeout, init_method, rank, world_size)
File "/usr/local/python3.10.13/lib/python3.10/site-packages/deepspeed/comm/torch.py", line 121, in init
self.init_process_group(backend, timeout, init_method, rank, world_size)
File "/usr/local/python3.10.13/lib/python3.10/site-packages/deepspeed/comm/torch.py", line 149, in init_process_group
torch.distributed.init_process_group(backend,
File "/usr/local/python3.10.13/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 74, in wrapper
func_return = func(*args, **kwargs)
File "/usr/local/python3.10.13/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1148, in init_process_group
default_pg, _ = _new_process_group_helper(
File "/usr/local/python3.10.13/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1268, in _new_process_group_helper
raise RuntimeError("Distributed package doesn't have NCCL built in")
RuntimeError: Distributed package doesn't have NCCL built in
[2025-05-26 13:54:17,563] [INFO] [comm.py:637:init_distributed] cdb=None
Traceback (most recent call last):
File "/home/gq/crossformer_qwen7b_llava/start_train.py", line 62, in
start_train()
File "/home/gq/crossformer_qwen7b_llava/start_train.py", line 56, in start_train
train()
File "/home/gq/crossformer_qwen7b_llava/llava_qwen7b_module/llava/train/train.py", line 981, in train
model_args, data_args, training_args = parser.parse_args_into_dataclasses()
File "/usr/local/python3.10.13/lib/python3.10/site-packages/transformers/hf_argparser.py", line 338, in parse_args_into_dataclasses
obj = dtype(**inputs)
File "", line 136, in init
File "/usr/local/python3.10.13/lib/python3.10/site-packages/transformers/training_args.py", line 1483, in post_init
and (self.device.type != "cuda")
File "/usr/local/python3.10.13/lib/python3.10/site-packages/transformers/training_args.py", line 1921, in device
return self._setup_devices
File "/usr/local/python3.10.13/lib/python3.10/site-packages/transformers/utils/generic.py", line 54, in get
cached = self.fget(obj)
File "/usr/local/python3.10.13/lib/python3.10/site-packages/transformers/training_args.py", line 1853, in _setup_devices
self.distributed_state = PartialState(timeout=timedelta(seconds=self.ddp_timeout))
File "/usr/local/python3.10.13/lib/python3.10/site-packages/accelerate/state.py", line 170, in init
dist.init_distributed(dist_backend=self.backend, auto_mpi_discovery=False, **kwargs)
File "/usr/local/python3.10.13/lib/python3.10/site-packages/deepspeed/comm/comm.py", line 670, in init_distributed
cdb = TorchBackend(dist_backend, timeout, init_method, rank, world_size)
File "/usr/local/python3.10.13/lib/python3.10/site-packages/deepspeed/comm/torch.py", line 121, in init
self.init_process_group(backend, timeout, init_method, rank, world_size)
File "/usr/local/python3.10.13/lib/python3.10/site-packages/deepspeed/comm/torch.py", line 149, in init_process_group
torch.distributed.init_process_group(backend,
File "/usr/local/python3.10.13/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 74, in wrapper
func_return = func(*args, **kwargs)
File "/usr/local/python3.10.13/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1148, in init_process_group
default_pg, _ = _new_process_group_helper(
File "/usr/local/python3.10.13/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1268, in _new_process_group_helper
raise RuntimeError("Distributed package doesn't have NCCL built in")
RuntimeError: Distributed package doesn't have NCCL built in
[2025-05-26 13:54:17,857] [INFO] [comm.py:637:init_distributed] cdb=None
Traceback (most recent call last):
File "/home/gq/crossformer_qwen7b_llava/start_train.py", line 62, in
start_train()
File "/home/gq/crossformer_qwen7b_llava/start_train.py", line 56, in start_train
train()
File "/home/gq/crossformer_qwen7b_llava/llava_qwen7b_module/llava/train/train.py", line 981, in train
model_args, data_args, training_args = parser.parse_args_into_dataclasses()
File "/usr/local/python3.10.13/lib/python3.10/site-packages/transformers/hf_argparser.py", line 338, in parse_args_into_dataclasses
obj = dtype(**inputs)
File "", line 136, in init
File "/usr/local/python3.10.13/lib/python3.10/site-packages/transformers/training_args.py", line 1483, in post_init
and (self.device.type != "cuda")
File "/usr/local/python3.10.13/lib/python3.10/site-packages/transformers/training_args.py", line 1921, in device
return self._setup_devices
File "/usr/local/python3.10.13/lib/python3.10/site-packages/transformers/utils/generic.py", line 54, in get
cached = self.fget(obj)
File "/usr/local/python3.10.13/lib/python3.10/site-packages/transformers/training_args.py", line 1853, in _setup_devices
self.distributed_state = PartialState(timeout=timedelta(seconds=self.ddp_timeout))
File "/usr/local/python3.10.13/lib/python3.10/site-packages/accelerate/state.py", line 170, in init
dist.init_distributed(dist_backend=self.backend, auto_mpi_discovery=False, **kwargs)
File "/usr/local/python3.10.13/lib/python3.10/site-packages/deepspeed/comm/comm.py", line 670, in init_distributed
cdb = TorchBackend(dist_backend, timeout, init_method, rank, world_size)
File "/usr/local/python3.10.13/lib/python3.10/site-packages/deepspeed/comm/torch.py", line 121, in init
self.init_process_group(backend, timeout, init_method, rank, world_size)
File "/usr/local/python3.10.13/lib/python3.10/site-packages/deepspeed/comm/torch.py", line 149, in init_process_group
torch.distributed.init_process_group(backend,
File "/usr/local/python3.10.13/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 74, in wrapper
func_return = func(*args, **kwargs)
File "/usr/local/python3.10.13/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1148, in init_process_group
default_pg, _ = _new_process_group_helper(
File "/usr/local/python3.10.13/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1268, in _new_process_group_helper
raise RuntimeError("Distributed package doesn't have NCCL built in")
RuntimeError: Distributed package doesn't have NCCL built in
[2025-05-26 13:54:17,901] [INFO] [comm.py:637:init_distributed] cdb=None
Traceback (most recent call last):
File "/home/gq/crossformer_qwen7b_llava/start_train.py", line 62, in
start_train()
File "/home/gq/crossformer_qwen7b_llava/start_train.py", line 56, in start_train
train()
File "/home/gq/crossformer_qwen7b_llava/llava_qwen7b_module/llava/train/train.py", line 981, in train
model_args, data_args, training_args = parser.parse_args_into_dataclasses()
File "/usr/local/python3.10.13/lib/python3.10/site-packages/transformers/hf_argparser.py", line 338, in parse_args_into_dataclasses
obj = dtype(**inputs)
File "", line 136, in init
File "/usr/local/python3.10.13/lib/python3.10/site-packages/transformers/training_args.py", line 1483, in post_init
and (self.device.type != "cuda")
File "/usr/local/python3.10.13/lib/python3.10/site-packages/transformers/training_args.py", line 1921, in device
return self._setup_devices
File "/usr/local/python3.10.13/lib/python3.10/site-packages/transformers/utils/generic.py", line 54, in get
cached = self.fget(obj)
File "/usr/local/python3.10.13/lib/python3.10/site-packages/transformers/training_args.py", line 1853, in _setup_devices
self.distributed_state = PartialState(timeout=timedelta(seconds=self.ddp_timeout))
File "/usr/local/python3.10.13/lib/python3.10/site-packages/accelerate/state.py", line 170, in init
dist.init_distributed(dist_backend=self.backend, auto_mpi_discovery=False, **kwargs)
File "/usr/local/python3.10.13/lib/python3.10/site-packages/deepspeed/comm/comm.py", line 670, in init_distributed
cdb = TorchBackend(dist_backend, timeout, init_method, rank, world_size)
File "/usr/local/python3.10.13/lib/python3.10/site-packages/deepspeed/comm/torch.py", line 121, in init
self.init_process_group(backend, timeout, init_method, rank, world_size)
File "/usr/local/python3.10.13/lib/python3.10/site-packages/deepspeed/comm/torch.py", line 149, in init_process_group
torch.distributed.init_process_group(backend,
File "/usr/local/python3.10.13/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 74, in wrapper
func_return = func(*args, **kwargs)
File "/usr/local/python3.10.13/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1148, in init_process_group
default_pg, _ = _new_process_group_helper(
File "/usr/local/python3.10.13/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1268, in _new_process_group_helper
raise RuntimeError("Distributed package doesn't have NCCL built in")
RuntimeError: Distributed package doesn't have NCCL built in
[2025-05-26 13:54:17,984] [INFO] [comm.py:637:init_distributed] cdb=None
Traceback (most recent call last):
File "/home/gq/crossformer_qwen7b_llava/start_train.py", line 62, in
start_train()
File "/home/gq/crossformer_qwen7b_llava/start_train.py", line 56, in start_train
train()
File "/home/gq/crossformer_qwen7b_llava/llava_qwen7b_module/llava/train/train.py", line 981, in train
model_args, data_args, training_args = parser.parse_args_into_dataclasses()
File "/usr/local/python3.10.13/lib/python3.10/site-packages/transformers/hf_argparser.py", line 338, in parse_args_into_dataclasses
obj = dtype(**inputs)
File "", line 136, in init
File "/usr/local/python3.10.13/lib/python3.10/site-packages/transformers/training_args.py", line 1483, in post_init
and (self.device.type != "cuda")
File "/usr/local/python3.10.13/lib/python3.10/site-packages/transformers/training_args.py", line 1921, in device
return self._setup_devices
File "/usr/local/python3.10.13/lib/python3.10/site-packages/transformers/utils/generic.py", line 54, in get
cached = self.fget(obj)
File "/usr/local/python3.10.13/lib/python3.10/site-packages/transformers/training_args.py", line 1853, in _setup_devices
self.distributed_state = PartialState(timeout=timedelta(seconds=self.ddp_timeout))
File "/usr/local/python3.10.13/lib/python3.10/site-packages/accelerate/state.py", line 170, in init
dist.init_distributed(dist_backend=self.backend, auto_mpi_discovery=False, **kwargs)
File "/usr/local/python3.10.13/lib/python3.10/site-packages/deepspeed/comm/comm.py", line 670, in init_distributed
cdb = TorchBackend(dist_backend, timeout, init_method, rank, world_size)
File "/usr/local/python3.10.13/lib/python3.10/site-packages/deepspeed/comm/torch.py", line 121, in init
self.init_process_group(backend, timeout, init_method, rank, world_size)
File "/usr/local/python3.10.13/lib/python3.10/site-packages/deepspeed/comm/torch.py", line 149, in init_process_group
torch.distributed.init_process_group(backend,
File "/usr/local/python3.10.13/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 74, in wrapper
func_return = func(*args, **kwargs)
File "/usr/local/python3.10.13/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1148, in init_process_group
default_pg, _ = _new_process_group_helper(
File "/usr/local/python3.10.13/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1268, in _new_process_group_helper
raise RuntimeError("Distributed package doesn't have NCCL built in")
RuntimeError: Distributed package doesn't have NCCL built in
Traceback (most recent call last):
File "/home/gq/crossformer_qwen7b_llava/start_train.py", line 62, in
start_train()
File "/home/gq/crossformer_qwen7b_llava/start_train.py", line 56, in start_train
train()
File "/home/gq/crossformer_qwen7b_llava/llava_qwen7b_module/llava/train/train.py", line 981, in train
model_args, data_args, training_args = parser.parse_args_into_dataclasses()
File "/usr/local/python3.10.13/lib/python3.10/site-packages/transformers/hf_argparser.py", line 338, in parse_args_into_dataclasses
obj = dtype(**inputs)
File "", line 136, in init
File "/usr/local/python3.10.13/lib/python3.10/site-packages/transformers/training_args.py", line 1483, in post_init
and (self.device.type != "cuda")
File "/usr/local/python3.10.13/lib/python3.10/site-packages/transformers/training_args.py", line 1921, in device
return self._setup_devices
File "/usr/local/python3.10.13/lib/python3.10/site-packages/transformers/utils/generic.py", line 54, in get
cached = self.fget(obj)
File "/usr/local/python3.10.13/lib/python3.10/site-packages/transformers/training_args.py", line 1853, in _setup_devices
self.distributed_state = PartialState(timeout=timedelta(seconds=self.ddp_timeout))
File "/usr/local/python3.10.13/lib/python3.10/site-packages/accelerate/state.py", line 170, in init
dist.init_distributed(dist_backend=self.backend, auto_mpi_discovery=False, **kwargs)
File "/usr/local/python3.10.13/lib/python3.10/site-packages/deepspeed/comm/comm.py", line 670, in init_distributed
cdb = TorchBackend(dist_backend, timeout, init_method, rank, world_size)
File "/usr/local/python3.10.13/lib/python3.10/site-packages/deepspeed/comm/torch.py", line 121, in init
self.init_process_group(backend, timeout, init_method, rank, world_size)
File "/usr/local/python3.10.13/lib/python3.10/site-packages/deepspeed/comm/torch.py", line 149, in init_process_group
torch.distributed.init_process_group(backend,
File "/usr/local/python3.10.13/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 74, in wrapper
func_return = func(*args, **kwargs)
File "/usr/local/python3.10.13/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1148, in init_process_group
default_pg, _ = _new_process_group_helper(
File "/usr/local/python3.10.13/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1268, in _new_process_group_helper
raise RuntimeError("Distributed package doesn't have NCCL built in")
RuntimeError: Distributed package doesn't have NCCL built in
[2025-05-26 13:54:18,696] [INFO] [comm.py:637:init_distributed] cdb=None
Traceback (most recent call last):
File "/home/gq/crossformer_qwen7b_llava/start_train.py", line 62, in
start_train()
File "/home/gq/crossformer_qwen7b_llava/start_train.py", line 56, in start_train
train()
File "/home/gq/crossformer_qwen7b_llava/llava_qwen7b_module/llava/train/train.py", line 981, in train
model_args, data_args, training_args = parser.parse_args_into_dataclasses()
File "/usr/local/python3.10.13/lib/python3.10/site-packages/transformers/hf_argparser.py", line 338, in parse_args_into_dataclasses
obj = dtype(**inputs)
File "", line 136, in init
File "/usr/local/python3.10.13/lib/python3.10/site-packages/transformers/training_args.py", line 1483, in post_init
and (self.device.type != "cuda")
File "/usr/local/python3.10.13/lib/python3.10/site-packages/transformers/training_args.py", line 1921, in device
return self._setup_devices
File "/usr/local/python3.10.13/lib/python3.10/site-packages/transformers/utils/generic.py", line 54, in get
cached = self.fget(obj)
File "/usr/local/python3.10.13/lib/python3.10/site-packages/transformers/training_args.py", line 1853, in _setup_devices
self.distributed_state = PartialState(timeout=timedelta(seconds=self.ddp_timeout))
File "/usr/local/python3.10.13/lib/python3.10/site-packages/accelerate/state.py", line 170, in init
dist.init_distributed(dist_backend=self.backend, auto_mpi_discovery=False, **kwargs)
File "/usr/local/python3.10.13/lib/python3.10/site-packages/deepspeed/comm/comm.py", line 670, in init_distributed
cdb = TorchBackend(dist_backend, timeout, init_method, rank, world_size)
File "/usr/local/python3.10.13/lib/python3.10/site-packages/deepspeed/comm/torch.py", line 121, in init
self.init_process_group(backend, timeout, init_method, rank, world_size)
File "/usr/local/python3.10.13/lib/python3.10/site-packages/deepspeed/comm/torch.py", line 149, in init_process_group
torch.distributed.init_process_group(backend,
File "/usr/local/python3.10.13/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 74, in wrapper
func_return = func(*args, **kwargs)
File "/usr/local/python3.10.13/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1148, in init_process_group
default_pg, _ = _new_process_group_helper(
File "/usr/local/python3.10.13/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1268, in _new_process_group_helper
raise RuntimeError("Distributed package doesn't have NCCL built in")
RuntimeError: Distributed package doesn't have NCCL built in
Traceback (most recent call last):
File "/home/gq/crossformer_qwen7b_llava/start_train.py", line 62, in
start_train()
File "/home/gq/crossformer_qwen7b_llava/start_train.py", line 56, in start_train
train()
File "/home/gq/crossformer_qwen7b_llava/llava_qwen7b_module/llava/train/train.py", line 981, in train
model_args, data_args, training_args = parser.parse_args_into_dataclasses()
File "/usr/local/python3.10.13/lib/python3.10/site-packages/transformers/hf_argparser.py", line 338, in parse_args_into_dataclasses
obj = dtype(**inputs)
File "", line 136, in init
File "/usr/local/python3.10.13/lib/python3.10/site-packages/transformers/training_args.py", line 1483, in post_init
and (self.device.type != "cuda")
File "/usr/local/python3.10.13/lib/python3.10/site-packages/transformers/training_args.py", line 1921, in device
return self._setup_devices
File "/usr/local/python3.10.13/lib/python3.10/site-packages/transformers/utils/generic.py", line 54, in get
cached = self.fget(obj)
File "/usr/local/python3.10.13/lib/python3.10/site-packages/transformers/training_args.py", line 1853, in _setup_devices
self.distributed_state = PartialState(timeout=timedelta(seconds=self.ddp_timeout))
File "/usr/local/python3.10.13/lib/python3.10/site-packages/accelerate/state.py", line 170, in init
dist.init_distributed(dist_backend=self.backend, auto_mpi_discovery=False, **kwargs)
File "/usr/local/python3.10.13/lib/python3.10/site-packages/deepspeed/comm/comm.py", line 670, in init_distributed
cdb = TorchBackend(dist_backend, timeout, init_method, rank, world_size)
File "/usr/local/python3.10.13/lib/python3.10/site-packages/deepspeed/comm/torch.py", line 121, in init
self.init_process_group(backend, timeout, init_method, rank, world_size)
File "/usr/local/python3.10.13/lib/python3.10/site-packages/deepspeed/comm/torch.py", line 149, in init_process_group
torch.distributed.init_process_group(backend,
File "/usr/local/python3.10.13/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 74, in wrapper
func_return = func(*args, **kwargs)
File "/usr/local/python3.10.13/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1148, in init_process_group
default_pg, _ = _new_process_group_helper(
File "/usr/local/python3.10.13/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1268, in _new_process_group_helper
raise RuntimeError("Distributed package doesn't have NCCL built in")
RuntimeError: Distributed package doesn't have NCCL built in
[2025-05-26 13:54:20,841] [INFO] [launch.py:316:sigkill_handler] Killing subprocess 442451
[2025-05-26 13:54:20,939] [INFO] [launch.py:316:sigkill_handler] Killing subprocess 442452
[2025-05-26 13:54:20,941] [INFO] [launch.py:316:sigkill_handler] Killing subprocess 442453
[2025-05-26 13:54:21,079] [INFO] [launch.py:316:sigkill_handler] Killing subprocess 442454
[2025-05-26 13:54:21,107] [INFO] [launch.py:316:sigkill_handler] Killing subprocess 442455
[2025-05-26 13:54:21,109] [INFO] [launch.py:316:sigkill_handler] Killing subprocess 442456
[2025-05-26 13:54:21,110] [INFO] [launch.py:316:sigkill_handler] Killing subprocess 442457
[2025-05-26 13:54:21,112] [INFO] [launch.py:316:sigkill_handler] Killing subprocess 442458
[2025-05-26 13:54:21,112] [ERROR] [launch.py:322:sigkill_handler] ['/usr/local/python3.10.13/bin/python3.10', '-u', 'start_train.py', '--local_rank=7'] exits with return code = 1

Start with deepspeed/comm/comm.py and deepspeed/comm/torch.py, then trace accelerator selection from real_accelerator.py:191 through the initialization shown in the log. Reproduce the failure with the reported start_train.py command and determine whether Huawei NPU training is supported or whether the backend configuration is incompatible; done means a confirmed cause and documented supported setup or limitation.