subprocess.CalledProcessError: Command '['/home/wushupei1997/anaconda3/envs/torch/bin/python', '-u','main2.py', '--local_rank=0', '--deepspeed_config', 'ds_config.json']' returned non-zero exit status1.

[RedHat]
this is my issue:

Emitting ninja build file /home/wushupei1997/.cache/torch_extensions/utils/build.ninja...
Building extension module utils...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[1/2] c++ -MMD -MF flatten_unflatten.o.d -DTORCH_EXTENSION_NAME=utils -DTORCH_API_INCLUDE_EXTENSION_H -isystem /home/wushupei1997/anaconda3/envs/torch/lib/python3.7/site-packages/torch/include -isystem/home/wushupei1997/anaconda3/envs/torch/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -isystem /home/wushupei1997/anaconda3/envs/torch/lib/python3.7/site-packages/torch/include/TH -isystem /home/wushupei1997/anaconda3/envs/torch/lib/python3.7/site-packages/torch/include/THC -isystem/home/wushupei1997/anaconda3/envs/torch/include/python3.7m -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -c /home/wushupei1997/anaconda3/envs/torch/lib/python3.7/site-packages/deepspeed-0.3.16-py3.7.egg/deepspeed/ops/csrc/utils/flatten_unflatten.cpp -o flatten_unflatten.o
FAILED: flatten_unflatten.o
c++ -MMD -MF flatten_unflatten.o.d -DTORCH_EXTENSION_NAME=utils -DTORCH_API_INCLUDE_EXTENSION_H -isystem /home/wushupei1997/anaconda3/envs/torch/lib/python3.7/site-packages/torch/include -isystem /home/wushupei1997/anaconda3/envs/torch/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -isystem /home/wushupei1997/anaconda3/envs/torch/lib/python3.7/site-packages/torch/include/TH -isystem/home/wushupei1997/anaconda3/envs/torch/lib/python3.7/site-packages/torch/include/THC -isystem /home/wushupei1997/anaconda3/envs/torch/include/python3.7m -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -c /home/wushupei1997/anaconda3/envs/torch/lib/python3.7/site-packages/deepspeed-0.3.16-py3.7.egg/deepspeed/ops/csrc/utils/flatten_unflatten.cpp -o flatten_unflatten.o
c++: error: unrecognized command line option ‘-std=c++14’
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
File "/home/wushupei1997/anaconda3/envs/torch/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1515, in _run_ninja_build
env=env)
File "/home/wushupei1997/anaconda3/envs/torch/lib/python3.7/subprocess.py", line 512, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "main2.py", line 187, in
main()
File "main2.py", line 108, in main
main_worker(args.local_rank, args.save_dir, args.model_name, args.batch_size, args.epochs, args.use_parallel, args.use_deep, args)
File "main2.py", line 172, in main_worker
verbose=2, validation_data=(val_input, val_target), args=args)
File "/home/wushupei1997/P3/src/models/basemodel.py", line 493, in fit
model, optim, _, _ = deepspeed.initialize(args=args, model=self.train(), optimizer=self.optim)
File "/home/wushupei1997/anaconda3/envs/torch/lib/python3.7/site-packages/deepspeed-0.3.16-py3.7.egg/deepspeed/init.py", line 130, in initialize
config_params=config_params)
File "/home/wushupei1997/anaconda3/envs/torch/lib/python3.7/site-packages/deepspeed-0.3.16-py3.7.egg/deepspeed/runtime/engine.py", line 198, in init
util_ops = UtilsBuilder().load()
File "/home/wushupei1997/anaconda3/envs/torch/lib/python3.7/site-packages/deepspeed-0.3.16-py3.7.egg/deepspeed/ops/op_builder/builder.py", line 215, in load
return self.jit_load(verbose)
File "/home/wushupei1997/anaconda3/envs/torch/lib/python3.7/site-packages/deepspeed-0.3.16-py3.7.egg/deepspeed/ops/op_builder/builder.py", line 252, in jit_load
verbose=verbose)
File "/home/wushupei1997/anaconda3/envs/torch/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 974, in load
keep_intermediates=keep_intermediates)
File "/home/wushupei1997/anaconda3/envs/torch/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1179, in _jit_compile
with_cuda=with_cuda)
File "/home/wushupei1997/anaconda3/envs/torch/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1279, in _write_ninja_file_and_build_library
error_prefix="Error building extension '{}'".format(name))
File "/home/wushupei1997/anaconda3/envs/torch/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1529, in _run_ninja_build
raise RuntimeError(message)
RuntimeError: Error building extension 'utils'
Killing subprocess 2716
Traceback (most recent call last):
File "/home/wushupei1997/anaconda3/envs/torch/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/home/wushupei1997/anaconda3/envs/torch/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/wushupei1997/anaconda3/envs/torch/lib/python3.7/site-packages/deepspeed-0.3.16-py3.7.egg/deepspeed/launcher/launch.py", line 171, in
main()
File "/home/wushupei1997/anaconda3/envs/torch/lib/python3.7/site-packages/deepspeed-0.3.16-py3.7.egg/deepspeed/launcher/launch.py", line 161, in main
sigkill_handler(signal.SIGTERM, None) # not coming back
File "/home/wushupei1997/anaconda3/envs/torch/lib/python3.7/site-packages/deepspeed-0.3.16-py3.7.egg/deepspeed/launcher/launch.py", line 139, in sigkill_handler
raise subprocess.CalledProcessError(returncode=last_return_code, cmd=cmd)
subprocess.CalledProcessError: Command '['/home/wushupei1997/anaconda3/envs/torch/bin/python', '-u','main2.py', '--local_rank=0', '--deepspeed_config', 'ds_config.json']' returned non-zero exit status1.

How can I solve this problem?

Start with the reported command in main2.py and the deepspeed.initialize call in src/models/basemodel.py, then inspect the torch C++ extension build output. Reproduce the failure around the c++ -std=c++14 invocation and review the two existing comments. Done should mean the environment-specific failure is explained or a project-scoped fix is identified.