Build error: error: identifier "cublasLtMatmulDescOpaque_t" is undefined

I am using PyTorch 1.8.1 in Cuda 10.2. whenever I try to build apex I get build error.

One of the error was "error: identifier "cublasLtMatmulDescOpaque_t" is undefined". How can I ass cublas?
Build Log:

2021-04-25T17:22:28,441 Using pip 21.1 from /home/partha9/py38_apex/lib/python3.8/site-packages/pip (python 3.8)
2021-04-25T17:22:28,444 Non-user install because user site-packages disabled
2021-04-25T17:22:28,524 Created temporary directory: /tmp/pip-ephem-wheel-cache-455uizkm
2021-04-25T17:22:28,525 Created temporary directory: /tmp/pip-req-tracker-f7d4hblw
2021-04-25T17:22:28,525 Initialized build tracking at /tmp/pip-req-tracker-f7d4hblw
2021-04-25T17:22:28,525 Created build tracker: /tmp/pip-req-tracker-f7d4hblw
2021-04-25T17:22:28,525 Entered build tracker: /tmp/pip-req-tracker-f7d4hblw
2021-04-25T17:22:28,525 Created temporary directory: /tmp/pip-install-1kyf02s9
2021-04-25T17:22:28,548 Looking in links: /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/nix/avx2, /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/nix/generic, /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/generic
2021-04-25T17:22:28,551 Processing /home/partha9/apex 2021-04-25T17:22:28,551   Created temporary directory: /tmp/pip-req-build-8zq6h56v
2021-04-25T17:22:28,551   DEPRECATION: A future pip version will change local packages to be built in-place without first copying to a temporary directory. We recommend you use --use-feature=in-tree-build to test your packages with this new behavior before it becomes the default.
2021-04-25T17:22:28,551    pip 21.3 will remove support for this functionality. You can find discussion regarding this at https://github.com/pypa/pip/issues/7555.
2021-04-25T17:22:28,990   Added file:///home/partha9/apex to build tracker '/tmp/pip-req-tracker-f7d4hblw'
2021-04-25T17:22:28,991     Running setup.py (path:/tmp/pip-req-build-8zq6h56v/setup.py) egg_info for package from file:///home/partha9/apex
2021-04-25T17:22:28,991     Created temporary directory: /tmp/pip-pip-egg-info-8ertna46
2021-04-25T17:22:28,991     Running command python setup.py egg_info
2021-04-25T17:22:30,938     No CUDA runtime is found, using CUDA_HOME='/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/cudacore/10.2.89'

2021-04-25T17:22:30,939     Warning: Torch did not find available GPUs on this system.
2021-04-25T17:22:30,939      If your intention is to cross-compile, this is not an error.
2021-04-25T17:22:30,939     By default, Apex will cross-compile for Pascal (compute capabilities 6.0, 6.1, 6.2),
2021-04-25T17:22:30,939     Volta (compute capability 7.0), Turing (compute capability 7.5),
2021-04-25T17:22:30,939     and, if the CUDA version is >= 11.0, Ampere (compute capability 8.0).
2021-04-25T17:22:30,939     If you wish to cross-compile for a single specific architecture,
2021-04-25T17:22:30,939     export TORCH_CUDA_ARCH_LIST="compute capability" before running setup.py.



2021-04-25T17:22:30,939     torch.__version__  = 1.8.1+cu102


2021-04-25T17:22:30,940     running egg_info
2021-04-25T17:22:30,940     creating /tmp/pip-pip-egg-info-8ertna46/apex.egg-info
2021-04-25T17:22:30,940     writing /tmp/pip-pip-egg-info-8ertna46/apex.egg-info/PKG-INFO
2021-04-25T17:22:30,940     writing dependency_links to /tmp/pip-pip-egg-info-8ertna46/apex.egg-info/dependency_links.txt
2021-04-25T17:22:30,940     writing top-level names to /tmp/pip-pip-egg-info-8ertna46/apex.egg-info/top_level.txt
2021-04-25T17:22:30,940     writing manifest file '/tmp/pip-pip-egg-info-8ertna46/apex.egg-info/SOURCES.txt'
2021-04-25T17:22:30,940     adding license file 'LICENSE' (matched pattern 'LICEN[CS]E*')
2021-04-25T17:22:30,940     reading manifest file '/tmp/pip-pip-egg-info-8ertna46/apex.egg-info/SOURCES.txt'
2021-04-25T17:22:30,941     writing manifest file '/tmp/pip-pip-egg-info-8ertna46/apex.egg-info/SOURCES.txt'
2021-04-25T17:22:30,942     /tmp/pip-req-build-8zq6h56v/setup.py:67: UserWarning: Option --pyprof not specified. Not installing PyProf dependencies!
2021-04-25T17:22:30,942       warnings.warn("Option --pyprof not specified. Not installing PyProf dependencies!")
2021-04-25T17:22:31,072   Source in /tmp/pip-req-build-8zq6h56v has version 0.1, which satisfies requirement apex==0.1 from file:///home/partha9/apex
2021-04-25T17:22:31,073   Removed apex==0.1 from file:///home/partha9/apex from build tracker '/tmp/pip-req-tracker-f7d4hblw'
2021-04-25T17:22:31,074 Created temporary directory: /tmp/pip-unpack-sf08mc58
2021-04-25T17:22:31,074 Skipping wheel build for apex, due to binaries being disabled for it.
2021-04-25T17:22:31,573 Installing collected packages: apex
2021-04-25T17:22:31,576   Created temporary directory: /tmp/pip-record-qf8wx_y_
2021-04-25T17:22:31,576     Running command /home/partha9/py38_apex/bin/python3 -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-8zq6h56v/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-8zq6h56v/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' --cpp_ext --cuda_ext install --record /tmp/pip-record-qf8wx_y_/install-record.txt --single-version-externally-managed --compile --install-headers /home/partha9/py38_apex/include/site/python3.8/apex
2021-04-25T17:22:33,327     No CUDA runtime is found, using CUDA_HOME='/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/cudacore/10.2.89'

2021-04-25T17:22:33,328     Warning: Torch did not find available GPUs on this system.
2021-04-25T17:22:33,328      If your intention is to cross-compile, this is not an error.
2021-04-25T17:22:33,328     By default, Apex will cross-compile for Pascal (compute capabilities 6.0, 6.1, 6.2),
2021-04-25T17:22:33,328     Volta (compute capability 7.0), Turing (compute capability 7.5),
2021-04-25T17:22:33,328     and, if the CUDA version is >= 11.0, Ampere (compute capability 8.0).
2021-04-25T17:22:33,328     If you wish to cross-compile for a single specific architecture,
2021-04-25T17:22:33,329     export TORCH_CUDA_ARCH_LIST="compute capability" before running setup.py.



2021-04-25T17:22:33,336     torch.__version__  = 1.8.1+cu102


2021-04-25T17:22:33,337     /tmp/pip-req-build-8zq6h56v/setup.py:67: UserWarning: Option --pyprof not specified. Not installing PyProf dependencies!
2021-04-25T17:22:33,337       warnings.warn("Option --pyprof not specified. Not installing PyProf dependencies!")

2021-04-25T17:22:33,344     Compiling cuda extensions with
2021-04-25T17:22:33,344     nvcc: NVIDIA (R) Cuda compiler driver
2021-04-25T17:22:33,344     Copyright (c) 2005-2019 NVIDIA Corporation
2021-04-25T17:22:33,344     Built on Wed_Oct_23_19:24:38_PDT_2019
2021-04-25T17:22:33,344     Cuda compilation tools, release 10.2, V10.2.89
2021-04-25T17:22:33,344     from /cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/cudacore/10.2.89/bin

2021-04-25T17:22:33,412     running install
2021-04-25T17:22:33,413     running build
2021-04-25T17:22:33,413     running build_ext
2021-04-25T17:22:33,444     building 'apex_C' extension
2021-04-25T17:22:33,444     creating /tmp/pip-req-build-8zq6h56v/build
2021-04-25T17:22:33,444     creating /tmp/pip-req-build-8zq6h56v/build/temp.linux-x86_64-3.8
2021-04-25T17:22:33,444     creating /tmp/pip-req-build-8zq6h56v/build/temp.linux-x86_64-3.8/csrc
2021-04-25T17:22:33,467     Emitting ninja build file /tmp/pip-req-build-8zq6h56v/build/temp.linux-x86_64-3.8/build.ninja...
2021-04-25T17:22:33,467     Compiling objects...
2021-04-25T17:22:33,467     Using envvar MAX_JOBS (2) as the number of workers...
2021-04-25T17:22:55,650     [1/1] c++ -MMD -MF /tmp/pip-req-build-8zq6h56v/build/temp.linux-x86_64-3.8/csrc/flatten_unflatten.o.d -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -O2 -ftree-vectorize -march=core-avx2 -fno-math-errno -fPIC -O2 -ftree-vectorize -march=core-avx2 -fno-math-errno -fPIC -fPIC -I/home/partha9/py38_apex/lib/python3.8/site-packages/torch/include -I/home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/TH -I/home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/THC -I/home/partha9/py38_apex/include -I/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Core/python/3.8.2/include/python3.8 -c -c /tmp/pip-req-build-8zq6h56v/csrc/flatten_unflatten.cpp -o /tmp/pip-req-build-8zq6h56v/build/temp.linux-x86_64-3.8/csrc/flatten_unflatten.o -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=apex_C -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
2021-04-25T17:22:55,651     In file included from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/ATen/Parallel.h:140:0,
2021-04-25T17:22:55,651                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/utils.h:3,
2021-04-25T17:22:55,651                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/nn/cloneable.h:5,
2021-04-25T17:22:55,651                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/nn.h:3,
2021-04-25T17:22:55,651                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/all.h:13,
2021-04-25T17:22:55,651                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/extension.h:4,
2021-04-25T17:22:55,651                      from /tmp/pip-req-build-8zq6h56v/csrc/flatten_unflatten.cpp:1:
2021-04-25T17:22:55,651     /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/ATen/ParallelOpenMP.h:83:0: warning: ignoring #pragma omp parallel [-Wunknown-pragmas]
2021-04-25T17:22:55,651      #pragma omp parallel for if ((end - begin) >= grain_size)

2021-04-25T17:22:55,651     In file included from /tmp/pip-req-build-8zq6h56v/csrc/flatten_unflatten.cpp:2:0:
2021-04-25T17:22:55,651     /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/utils/tensor_flatten.h: In member function ‘at::DeprecatedTypeProperties& torch::utils::TensorGroup::type()’:
2021-04-25T17:22:55,651     /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/utils/tensor_flatten.h:44:28: warning: ‘at::DeprecatedTypeProperties& at::Tensor::type() const’ is deprecated: Tensor.type() is deprecated. Instead use Tensor.options(), which in many cases (e.g. in a constructor) is a drop-in replacement. If you were using data from type(), that is now available from Tensor itself, so instead of tensor.type().scalar_type(), use tensor.scalar_type() instead and instead of tensor.type().backend() use tensor.device(). [-Wdeprecated-declarations]
2021-04-25T17:22:55,651          return tensors[0].type();
2021-04-25T17:22:55,651                                 ^
2021-04-25T17:22:55,652     In file included from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/ATen/Tensor.h:3:0,
2021-04-25T17:22:55,652                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/ATen/Context.h:4,
2021-04-25T17:22:55,652                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/ATen/ATen.h:9,
2021-04-25T17:22:55,652                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/types.h:3,
2021-04-25T17:22:55,652                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader_options.h:4,
2021-04-25T17:22:55,652                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/base.h:3,
2021-04-25T17:22:55,652                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/stateful.h:3,
2021-04-25T17:22:55,652                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader.h:3,
2021-04-25T17:22:55,652                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data.h:3,
2021-04-25T17:22:55,652                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/all.h:8,
2021-04-25T17:22:55,652                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/extension.h:4,
2021-04-25T17:22:55,652                      from /tmp/pip-req-build-8zq6h56v/csrc/flatten_unflatten.cpp:1:
2021-04-25T17:22:55,652     /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/ATen/core/TensorBody.h:303:30: note: declared here
2021-04-25T17:22:55,652        DeprecatedTypeProperties & type() const {
2021-04-25T17:22:55,652                                   ^~~~
2021-04-25T17:22:55,653     creating build/lib.linux-x86_64-3.8
2021-04-25T17:22:55,653     g++ -shared -L/cvmfs/soft.computecanada.ca/easybuild/software/2020/Core/libffi/3.3/lib64 -L/cvmfs/soft.computecanada.ca/easybuild/software/2020/Core/libffi/3.3/lib -L/cvmfs/soft.computecanada.ca/easybuild/software/2020/Core/gcccore/9.3.0/lib64 -L/cvmfs/soft.computecanada.ca/easybuild/software/2020/Core/gcccore/9.3.0/lib -L/cvmfs/soft.computecanada.ca/easybuild/software/2020/Core/libffi/3.3/lib64 -L/cvmfs/soft.computecanada.ca/easybuild/software/2020/Core/libffi/3.3/lib -L/cvmfs/soft.computecanada.ca/easybuild/software/2020/Core/gcccore/9.3.0/lib64 -L/cvmfs/soft.computecanada.ca/easybuild/software/2020/Core/gcccore/9.3.0/lib /tmp/pip-req-build-8zq6h56v/build/temp.linux-x86_64-3.8/csrc/flatten_unflatten.o -L/home/partha9/py38_apex/lib/python3.8/site-packages/torch/lib -L/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Core/python/3.8.2/lib -lc10 -ltorch -ltorch_cpu -ltorch_python -o build/lib.linux-x86_64-3.8/apex_C.cpython-38-x86_64-linux-gnu.so
2021-04-25T17:22:56,233     building 'amp_C' extension
2021-04-25T17:22:56,265     Emitting ninja build file /tmp/pip-req-build-8zq6h56v/build/temp.linux-x86_64-3.8/build.ninja...
2021-04-25T17:22:56,266     Compiling objects...
2021-04-25T17:22:56,266     Using envvar MAX_JOBS (2) as the number of workers...
2021-04-25T17:23:20,614     [1/11] c++ -MMD -MF /tmp/pip-req-build-8zq6h56v/build/temp.linux-x86_64-3.8/csrc/amp_C_frontend.o.d -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -O2 -ftree-vectorize -march=core-avx2 -fno-math-errno -fPIC -O2 -ftree-vectorize -march=core-avx2 -fno-math-errno -fPIC -fPIC -I/home/partha9/py38_apex/lib/python3.8/site-packages/torch/include -I/home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/TH -I/home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/THC -I/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/cudacore/10.2.89/include -I/home/partha9/py38_apex/include -I/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Core/python/3.8.2/include/python3.8 -c -c /tmp/pip-req-build-8zq6h56v/csrc/amp_C_frontend.cpp -o /tmp/pip-req-build-8zq6h56v/build/temp.linux-x86_64-3.8/csrc/amp_C_frontend.o -O3 -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=amp_C -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
2021-04-25T17:23:20,615     In file included from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/ATen/Parallel.h:140:0,
2021-04-25T17:23:20,615                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/utils.h:3,
2021-04-25T17:23:20,615                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/nn/cloneable.h:5,
2021-04-25T17:23:20,615                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/nn.h:3,
2021-04-25T17:23:20,615                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/all.h:13,
2021-04-25T17:23:20,615                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/extension.h:4,
2021-04-25T17:23:20,615                      from /tmp/pip-req-build-8zq6h56v/csrc/amp_C_frontend.cpp:1:
2021-04-25T17:23:20,615     /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/ATen/ParallelOpenMP.h:83:0: warning: ignoring #pragma omp parallel [-Wunknown-pragmas]
2021-04-25T17:23:20,615      #pragma omp parallel for if ((end - begin) >= grain_size)

2021-04-25T17:23:38,331     [2/11] /cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/cudacore/10.2.89/bin/nvcc --generate-dependencies-with-compile --dependency-output /tmp/pip-req-build-8zq6h56v/build/temp.linux-x86_64-3.8/csrc/multi_tensor_scale_kernel.o.d -I/home/partha9/py38_apex/lib/python3.8/site-packages/torch/include -I/home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/TH -I/home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/THC -I/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/cudacore/10.2.89/include -I/home/partha9/py38_apex/include -I/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Core/python/3.8.2/include/python3.8 -c -c /tmp/pip-req-build-8zq6h56v/csrc/multi_tensor_scale_kernel.cu -o /tmp/pip-req-build-8zq6h56v/build/temp.linux-x86_64-3.8/csrc/multi_tensor_scale_kernel.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -lineinfo -O3 --use_fast_math -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=amp_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_60,code=sm_60 -gencode=arch=compute_61,code=sm_61 -gencode=arch=compute_62,code=sm_62 -gencode=arch=compute_70,code=sm_70 -gencode=arch=compute_75,code=sm_75 -std=c++14
2021-04-25T17:24:01,187     [3/11] /cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/cudacore/10.2.89/bin/nvcc --generate-dependencies-with-compile --dependency-output /tmp/pip-req-build-8zq6h56v/build/temp.linux-x86_64-3.8/csrc/multi_tensor_sgd_kernel.o.d -I/home/partha9/py38_apex/lib/python3.8/site-packages/torch/include -I/home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/TH -I/home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/THC -I/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/cudacore/10.2.89/include -I/home/partha9/py38_apex/include -I/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Core/python/3.8.2/include/python3.8 -c -c /tmp/pip-req-build-8zq6h56v/csrc/multi_tensor_sgd_kernel.cu -o /tmp/pip-req-build-8zq6h56v/build/temp.linux-x86_64-3.8/csrc/multi_tensor_sgd_kernel.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -lineinfo -O3 --use_fast_math -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=amp_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_60,code=sm_60 -gencode=arch=compute_61,code=sm_61 -gencode=arch=compute_62,code=sm_62 -gencode=arch=compute_70,code=sm_70 -gencode=arch=compute_75,code=sm_75 -std=c++14
2021-04-25T17:24:22,245     [4/11] /cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/cudacore/10.2.89/bin/nvcc --generate-dependencies-with-compile --dependency-output /tmp/pip-req-build-8zq6h56v/build/temp.linux-x86_64-3.8/csrc/multi_tensor_axpby_kernel.o.d -I/home/partha9/py38_apex/lib/python3.8/site-packages/torch/include -I/home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/TH -I/home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/THC -I/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/cudacore/10.2.89/include -I/home/partha9/py38_apex/include -I/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Core/python/3.8.2/include/python3.8 -c -c /tmp/pip-req-build-8zq6h56v/csrc/multi_tensor_axpby_kernel.cu -o /tmp/pip-req-build-8zq6h56v/build/temp.linux-x86_64-3.8/csrc/multi_tensor_axpby_kernel.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -lineinfo -O3 --use_fast_math -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=amp_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_60,code=sm_60 -gencode=arch=compute_61,code=sm_61 -gencode=arch=compute_62,code=sm_62 -gencode=arch=compute_70,code=sm_70 -gencode=arch=compute_75,code=sm_75 -std=c++14
2021-04-25T17:24:44,653     [5/11] /cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/cudacore/10.2.89/bin/nvcc --generate-dependencies-with-compile --dependency-output /tmp/pip-req-build-8zq6h56v/build/temp.linux-x86_64-3.8/csrc/multi_tensor_l2norm_kernel.o.d -I/home/partha9/py38_apex/lib/python3.8/site-packages/torch/include -I/home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/TH -I/home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/THC -I/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/cudacore/10.2.89/include -I/home/partha9/py38_apex/include -I/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Core/python/3.8.2/include/python3.8 -c -c /tmp/pip-req-build-8zq6h56v/csrc/multi_tensor_l2norm_kernel.cu -o /tmp/pip-req-build-8zq6h56v/build/temp.linux-x86_64-3.8/csrc/multi_tensor_l2norm_kernel.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -lineinfo -O3 --use_fast_math -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=amp_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_60,code=sm_60 -gencode=arch=compute_61,code=sm_61 -gencode=arch=compute_62,code=sm_62 -gencode=arch=compute_70,code=sm_70 -gencode=arch=compute_75,code=sm_75 -std=c++14
2021-04-25T17:25:05,723     [6/11] /cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/cudacore/10.2.89/bin/nvcc --generate-dependencies-with-compile --dependency-output /tmp/pip-req-build-8zq6h56v/build/temp.linux-x86_64-3.8/csrc/multi_tensor_lamb_stage_1.o.d -I/home/partha9/py38_apex/lib/python3.8/site-packages/torch/include -I/home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/TH -I/home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/THC -I/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/cudacore/10.2.89/include -I/home/partha9/py38_apex/include -I/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Core/python/3.8.2/include/python3.8 -c -c /tmp/pip-req-build-8zq6h56v/csrc/multi_tensor_lamb_stage_1.cu -o /tmp/pip-req-build-8zq6h56v/build/temp.linux-x86_64-3.8/csrc/multi_tensor_lamb_stage_1.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -lineinfo -O3 --use_fast_math -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=amp_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_60,code=sm_60 -gencode=arch=compute_61,code=sm_61 -gencode=arch=compute_62,code=sm_62 -gencode=arch=compute_70,code=sm_70 -gencode=arch=compute_75,code=sm_75 -std=c++14
2021-04-25T17:25:26,412     [7/11] /cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/cudacore/10.2.89/bin/nvcc --generate-dependencies-with-compile --dependency-output /tmp/pip-req-build-8zq6h56v/build/temp.linux-x86_64-3.8/csrc/multi_tensor_lamb_stage_2.o.d -I/home/partha9/py38_apex/lib/python3.8/site-packages/torch/include -I/home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/TH -I/home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/THC -I/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/cudacore/10.2.89/include -I/home/partha9/py38_apex/include -I/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Core/python/3.8.2/include/python3.8 -c -c /tmp/pip-req-build-8zq6h56v/csrc/multi_tensor_lamb_stage_2.cu -o /tmp/pip-req-build-8zq6h56v/build/temp.linux-x86_64-3.8/csrc/multi_tensor_lamb_stage_2.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -lineinfo -O3 --use_fast_math -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=amp_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_60,code=sm_60 -gencode=arch=compute_61,code=sm_61 -gencode=arch=compute_62,code=sm_62 -gencode=arch=compute_70,code=sm_70 -gencode=arch=compute_75,code=sm_75 -std=c++14
2021-04-25T17:25:49,576     [8/11] /cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/cudacore/10.2.89/bin/nvcc --generate-dependencies-with-compile --dependency-output /tmp/pip-req-build-8zq6h56v/build/temp.linux-x86_64-3.8/csrc/multi_tensor_adam.o.d -I/home/partha9/py38_apex/lib/python3.8/site-packages/torch/include -I/home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/TH -I/home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/THC -I/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/cudacore/10.2.89/include -I/home/partha9/py38_apex/include -I/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Core/python/3.8.2/include/python3.8 -c -c /tmp/pip-req-build-8zq6h56v/csrc/multi_tensor_adam.cu -o /tmp/pip-req-build-8zq6h56v/build/temp.linux-x86_64-3.8/csrc/multi_tensor_adam.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -lineinfo -O3 --use_fast_math -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=amp_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_60,code=sm_60 -gencode=arch=compute_61,code=sm_61 -gencode=arch=compute_62,code=sm_62 -gencode=arch=compute_70,code=sm_70 -gencode=arch=compute_75,code=sm_75 -std=c++14
2021-04-25T17:26:09,328     [9/11] /cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/cudacore/10.2.89/bin/nvcc --generate-dependencies-with-compile --dependency-output /tmp/pip-req-build-8zq6h56v/build/temp.linux-x86_64-3.8/csrc/multi_tensor_adagrad.o.d -I/home/partha9/py38_apex/lib/python3.8/site-packages/torch/include -I/home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/TH -I/home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/THC -I/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/cudacore/10.2.89/include -I/home/partha9/py38_apex/include -I/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Core/python/3.8.2/include/python3.8 -c -c /tmp/pip-req-build-8zq6h56v/csrc/multi_tensor_adagrad.cu -o /tmp/pip-req-build-8zq6h56v/build/temp.linux-x86_64-3.8/csrc/multi_tensor_adagrad.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -lineinfo -O3 --use_fast_math -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=amp_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_60,code=sm_60 -gencode=arch=compute_61,code=sm_61 -gencode=arch=compute_62,code=sm_62 -gencode=arch=compute_70,code=sm_70 -gencode=arch=compute_75,code=sm_75 -std=c++14
2021-04-25T17:26:35,226     [10/11] /cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/cudacore/10.2.89/bin/nvcc --generate-dependencies-with-compile --dependency-output /tmp/pip-req-build-8zq6h56v/build/temp.linux-x86_64-3.8/csrc/multi_tensor_novograd.o.d -I/home/partha9/py38_apex/lib/python3.8/site-packages/torch/include -I/home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/TH -I/home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/THC -I/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/cudacore/10.2.89/include -I/home/partha9/py38_apex/include -I/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Core/python/3.8.2/include/python3.8 -c -c /tmp/pip-req-build-8zq6h56v/csrc/multi_tensor_novograd.cu -o /tmp/pip-req-build-8zq6h56v/build/temp.linux-x86_64-3.8/csrc/multi_tensor_novograd.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -lineinfo -O3 --use_fast_math -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=amp_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_60,code=sm_60 -gencode=arch=compute_61,code=sm_61 -gencode=arch=compute_62,code=sm_62 -gencode=arch=compute_70,code=sm_70 -gencode=arch=compute_75,code=sm_75 -std=c++14
2021-04-25T17:26:55,515     [11/11] /cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/cudacore/10.2.89/bin/nvcc --generate-dependencies-with-compile --dependency-output /tmp/pip-req-build-8zq6h56v/build/temp.linux-x86_64-3.8/csrc/multi_tensor_lamb.o.d -I/home/partha9/py38_apex/lib/python3.8/site-packages/torch/include -I/home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/TH -I/home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/THC -I/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/cudacore/10.2.89/include -I/home/partha9/py38_apex/include -I/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Core/python/3.8.2/include/python3.8 -c -c /tmp/pip-req-build-8zq6h56v/csrc/multi_tensor_lamb.cu -o /tmp/pip-req-build-8zq6h56v/build/temp.linux-x86_64-3.8/csrc/multi_tensor_lamb.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -lineinfo -O3 --use_fast_math -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=amp_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_60,code=sm_60 -gencode=arch=compute_61,code=sm_61 -gencode=arch=compute_62,code=sm_62 -gencode=arch=compute_70,code=sm_70 -gencode=arch=compute_75,code=sm_75 -std=c++14
2021-04-25T17:26:55,515     g++ -shared -L/cvmfs/soft.computecanada.ca/easybuild/software/2020/Core/libffi/3.3/lib64 -L/cvmfs/soft.computecanada.ca/easybuild/software/2020/Core/libffi/3.3/lib -L/cvmfs/soft.computecanada.ca/easybuild/software/2020/Core/gcccore/9.3.0/lib64 -L/cvmfs/soft.computecanada.ca/easybuild/software/2020/Core/gcccore/9.3.0/lib -L/cvmfs/soft.computecanada.ca/easybuild/software/2020/Core/libffi/3.3/lib64 -L/cvmfs/soft.computecanada.ca/easybuild/software/2020/Core/libffi/3.3/lib -L/cvmfs/soft.computecanada.ca/easybuild/software/2020/Core/gcccore/9.3.0/lib64 -L/cvmfs/soft.computecanada.ca/easybuild/software/2020/Core/gcccore/9.3.0/lib /tmp/pip-req-build-8zq6h56v/build/temp.linux-x86_64-3.8/csrc/amp_C_frontend.o /tmp/pip-req-build-8zq6h56v/build/temp.linux-x86_64-3.8/csrc/multi_tensor_sgd_kernel.o /tmp/pip-req-build-8zq6h56v/build/temp.linux-x86_64-3.8/csrc/multi_tensor_scale_kernel.o /tmp/pip-req-build-8zq6h56v/build/temp.linux-x86_64-3.8/csrc/multi_tensor_axpby_kernel.o /tmp/pip-req-build-8zq6h56v/build/temp.linux-x86_64-3.8/csrc/multi_tensor_l2norm_kernel.o /tmp/pip-req-build-8zq6h56v/build/temp.linux-x86_64-3.8/csrc/multi_tensor_lamb_stage_1.o /tmp/pip-req-build-8zq6h56v/build/temp.linux-x86_64-3.8/csrc/multi_tensor_lamb_stage_2.o /tmp/pip-req-build-8zq6h56v/build/temp.linux-x86_64-3.8/csrc/multi_tensor_adam.o /tmp/pip-req-build-8zq6h56v/build/temp.linux-x86_64-3.8/csrc/multi_tensor_adagrad.o /tmp/pip-req-build-8zq6h56v/build/temp.linux-x86_64-3.8/csrc/multi_tensor_novograd.o /tmp/pip-req-build-8zq6h56v/build/temp.linux-x86_64-3.8/csrc/multi_tensor_lamb.o -L/home/partha9/py38_apex/lib/python3.8/site-packages/torch/lib -L/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/cudacore/10.2.89/lib64 -L/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Core/python/3.8.2/lib -lc10 -ltorch -ltorch_cpu -ltorch_python -lcudart -lc10_cuda -ltorch_cuda -o build/lib.linux-x86_64-3.8/amp_C.cpython-38-x86_64-linux-gnu.so
2021-04-25T17:26:56,151     building 'syncbn' extension
2021-04-25T17:26:56,180     Emitting ninja build file /tmp/pip-req-build-8zq6h56v/build/temp.linux-x86_64-3.8/build.ninja...
2021-04-25T17:26:56,180     Compiling objects...
2021-04-25T17:26:56,180     Using envvar MAX_JOBS (2) as the number of workers...
2021-04-25T17:27:24,039     [1/2] c++ -MMD -MF /tmp/pip-req-build-8zq6h56v/build/temp.linux-x86_64-3.8/csrc/syncbn.o.d -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -O2 -ftree-vectorize -march=core-avx2 -fno-math-errno -fPIC -O2 -ftree-vectorize -march=core-avx2 -fno-math-errno -fPIC -fPIC -I/home/partha9/py38_apex/lib/python3.8/site-packages/torch/include -I/home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/TH -I/home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/THC -I/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/cudacore/10.2.89/include -I/home/partha9/py38_apex/include -I/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Core/python/3.8.2/include/python3.8 -c -c /tmp/pip-req-build-8zq6h56v/csrc/syncbn.cpp -o /tmp/pip-req-build-8zq6h56v/build/temp.linux-x86_64-3.8/csrc/syncbn.o -O3 -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=syncbn -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
2021-04-25T17:27:24,040     In file included from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/ATen/Parallel.h:140:0,
2021-04-25T17:27:24,040                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/utils.h:3,
2021-04-25T17:27:24,040                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/nn/cloneable.h:5,
2021-04-25T17:27:24,040                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/nn.h:3,
2021-04-25T17:27:24,040                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/all.h:13,
2021-04-25T17:27:24,040                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/extension.h:4,
2021-04-25T17:27:24,041                      from /tmp/pip-req-build-8zq6h56v/csrc/syncbn.cpp:1:
2021-04-25T17:27:24,041     /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/ATen/ParallelOpenMP.h:83:0: warning: ignoring #pragma omp parallel [-Wunknown-pragmas]
2021-04-25T17:27:24,041      #pragma omp parallel for if ((end - begin) >= grain_size)

2021-04-25T17:27:44,376     [2/2] /cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/cudacore/10.2.89/bin/nvcc --generate-dependencies-with-compile --dependency-output /tmp/pip-req-build-8zq6h56v/build/temp.linux-x86_64-3.8/csrc/welford.o.d -I/home/partha9/py38_apex/lib/python3.8/site-packages/torch/include -I/home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/TH -I/home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/THC -I/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/cudacore/10.2.89/include -I/home/partha9/py38_apex/include -I/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Core/python/3.8.2/include/python3.8 -c -c /tmp/pip-req-build-8zq6h56v/csrc/welford.cu -o /tmp/pip-req-build-8zq6h56v/build/temp.linux-x86_64-3.8/csrc/welford.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -O3 -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=syncbn -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_60,code=sm_60 -gencode=arch=compute_61,code=sm_61 -gencode=arch=compute_62,code=sm_62 -gencode=arch=compute_70,code=sm_70 -gencode=arch=compute_75,code=sm_75 -std=c++14
2021-04-25T17:27:44,379     g++ -shared -L/cvmfs/soft.computecanada.ca/easybuild/software/2020/Core/libffi/3.3/lib64 -L/cvmfs/soft.computecanada.ca/easybuild/software/2020/Core/libffi/3.3/lib -L/cvmfs/soft.computecanada.ca/easybuild/software/2020/Core/gcccore/9.3.0/lib64 -L/cvmfs/soft.computecanada.ca/easybuild/software/2020/Core/gcccore/9.3.0/lib -L/cvmfs/soft.computecanada.ca/easybuild/software/2020/Core/libffi/3.3/lib64 -L/cvmfs/soft.computecanada.ca/easybuild/software/2020/Core/libffi/3.3/lib -L/cvmfs/soft.computecanada.ca/easybuild/software/2020/Core/gcccore/9.3.0/lib64 -L/cvmfs/soft.computecanada.ca/easybuild/software/2020/Core/gcccore/9.3.0/lib /tmp/pip-req-build-8zq6h56v/build/temp.linux-x86_64-3.8/csrc/syncbn.o /tmp/pip-req-build-8zq6h56v/build/temp.linux-x86_64-3.8/csrc/welford.o -L/home/partha9/py38_apex/lib/python3.8/site-packages/torch/lib -L/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/cudacore/10.2.89/lib64 -L/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Core/python/3.8.2/lib -lc10 -ltorch -ltorch_cpu -ltorch_python -lcudart -lc10_cuda -ltorch_cuda -o build/lib.linux-x86_64-3.8/syncbn.cpython-38-x86_64-linux-gnu.so
2021-04-25T17:27:45,002     building 'fused_layer_norm_cuda' extension
2021-04-25T17:27:45,027     Emitting ninja build file /tmp/pip-req-build-8zq6h56v/build/temp.linux-x86_64-3.8/build.ninja...
2021-04-25T17:27:45,027     Compiling objects...
2021-04-25T17:27:45,027     Using envvar MAX_JOBS (2) as the number of workers...
2021-04-25T17:28:09,036     [1/2] c++ -MMD -MF /tmp/pip-req-build-8zq6h56v/build/temp.linux-x86_64-3.8/csrc/layer_norm_cuda.o.d -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -O2 -ftree-vectorize -march=core-avx2 -fno-math-errno -fPIC -O2 -ftree-vectorize -march=core-avx2 -fno-math-errno -fPIC -fPIC -I/home/partha9/py38_apex/lib/python3.8/site-packages/torch/include -I/home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/TH -I/home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/THC -I/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/cudacore/10.2.89/include -I/home/partha9/py38_apex/include -I/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Core/python/3.8.2/include/python3.8 -c -c /tmp/pip-req-build-8zq6h56v/csrc/layer_norm_cuda.cpp -o /tmp/pip-req-build-8zq6h56v/build/temp.linux-x86_64-3.8/csrc/layer_norm_cuda.o -O3 -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=fused_layer_norm_cuda -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
2021-04-25T17:28:09,037     In file included from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/ATen/Parallel.h:140:0,
2021-04-25T17:28:09,037                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/utils.h:3,
2021-04-25T17:28:09,037                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/nn/cloneable.h:5,
2021-04-25T17:28:09,037                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/nn.h:3,
2021-04-25T17:28:09,038                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/all.h:13,
2021-04-25T17:28:09,038                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/extension.h:4,
2021-04-25T17:28:09,038                      from /tmp/pip-req-build-8zq6h56v/csrc/layer_norm_cuda.cpp:1:
2021-04-25T17:28:09,038     /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/ATen/ParallelOpenMP.h:83:0: warning: ignoring #pragma omp parallel [-Wunknown-pragmas]
2021-04-25T17:28:09,038      #pragma omp parallel for if ((end - begin) >= grain_size)

2021-04-25T17:28:09,038     In file included from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/c10/core/DeviceType.h:8:0,
2021-04-25T17:28:09,038                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/c10/core/Device.h:3,
2021-04-25T17:28:09,038                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/c10/core/Allocator.h:6,
2021-04-25T17:28:09,038                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/ATen/ATen.h:7,
2021-04-25T17:28:09,038                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/types.h:3,
2021-04-25T17:28:09,038                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader_options.h:4,
2021-04-25T17:28:09,038                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/base.h:3,
2021-04-25T17:28:09,038                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/stateful.h:3,
2021-04-25T17:28:09,038                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader.h:3,
2021-04-25T17:28:09,038                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data.h:3,
2021-04-25T17:28:09,038                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/all.h:8,
2021-04-25T17:28:09,038                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/extension.h:4,
2021-04-25T17:28:09,039                      from /tmp/pip-req-build-8zq6h56v/csrc/layer_norm_cuda.cpp:1:
2021-04-25T17:28:09,039     /tmp/pip-req-build-8zq6h56v/csrc/layer_norm_cuda.cpp: In function ‘std::vector<at::Tensor> layer_norm(at::Tensor, c10::IntArrayRef, double)’:
2021-04-25T17:28:09,039     /tmp/pip-req-build-8zq6h56v/csrc/layer_norm_cuda.cpp:117:42: warning: ‘at::DeprecatedTypeProperties& at::Tensor::type() const’ is deprecated: Tensor.type() is deprecated. Instead use Tensor.options(), which in many cases (e.g. in a constructor) is a drop-in replacement. If you were using data from type(), that is now available from Tensor itself, so instead of tensor.type().scalar_type(), use tensor.scalar_type() instead and instead of tensor.type().backend() use tensor.device(). [-Wdeprecated-declarations]
2021-04-25T17:28:09,039      #define CHECK_CUDA(x) TORCH_CHECK(x.type().is_cuda(), #x " must be a CUDA tensor")
2021-04-25T17:28:09,039                                               ^
2021-04-25T17:28:09,039     /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/c10/macros/Macros.h:173:65: note: in definition of macro ‘C10_UNLIKELY’
2021-04-25T17:28:09,039      #define C10_UNLIKELY(expr)  (__builtin_expect(static_cast<bool>(expr), 0))
2021-04-25T17:28:09,039                                                                      ^~~~
2021-04-25T17:28:09,039     /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/c10/util/Exception.h:362:7: note: in expansion of macro ‘C10_UNLIKELY_OR_CONST’
2021-04-25T17:28:09,039        if (C10_UNLIKELY_OR_CONST(!(cond))) {                                 \
2021-04-25T17:28:09,039            ^~~~~~~~~~~~~~~~~~~~~
2021-04-25T17:28:09,039     /tmp/pip-req-build-8zq6h56v/csrc/layer_norm_cuda.cpp:117:23: note: in expansion of macro ‘TORCH_CHECK’
2021-04-25T17:28:09,039      #define CHECK_CUDA(x) TORCH_CHECK(x.type().is_cuda(), #x " must be a CUDA tensor")
2021-04-25T17:28:09,039                            ^~~~~~~~~~~
2021-04-25T17:28:09,039     /tmp/pip-req-build-8zq6h56v/csrc/layer_norm_cuda.cpp:119:24: note: in expansion of macro ‘CHECK_CUDA’
2021-04-25T17:28:09,040      #define CHECK_INPUT(x) CHECK_CUDA(x); CHECK_CONTIGUOUS(x)
2021-04-25T17:28:09,040                             ^~~~~~~~~~
2021-04-25T17:28:09,040     /tmp/pip-req-build-8zq6h56v/csrc/layer_norm_cuda.cpp:129:3: note: in expansion of macro ‘CHECK_INPUT’
2021-04-25T17:28:09,040        CHECK_INPUT(input);
2021-04-25T17:28:09,040        ^
2021-04-25T17:28:09,040     In file included from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/ATen/Tensor.h:3:0,
2021-04-25T17:28:09,040                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/ATen/Context.h:4,
2021-04-25T17:28:09,040                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/ATen/ATen.h:9,
2021-04-25T17:28:09,040                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/types.h:3,
2021-04-25T17:28:09,040                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader_options.h:4,
2021-04-25T17:28:09,040                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/base.h:3,
2021-04-25T17:28:09,040                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/stateful.h:3,
2021-04-25T17:28:09,040                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader.h:3,
2021-04-25T17:28:09,040                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data.h:3,
2021-04-25T17:28:09,040                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/all.h:8,
2021-04-25T17:28:09,040                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/extension.h:4,
2021-04-25T17:28:09,040                      from /tmp/pip-req-build-8zq6h56v/csrc/layer_norm_cuda.cpp:1:
2021-04-25T17:28:09,040     /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/ATen/core/TensorBody.h:303:30: note: declared here
2021-04-25T17:28:09,040        DeprecatedTypeProperties & type() const {
2021-04-25T17:28:09,041                                   ^~~~
2021-04-25T17:28:09,041     In file included from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/c10/core/DeviceType.h:8:0,
2021-04-25T17:28:09,041                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/c10/core/Device.h:3,
2021-04-25T17:28:09,041                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/c10/core/Allocator.h:6,
2021-04-25T17:28:09,041                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/ATen/ATen.h:7,
2021-04-25T17:28:09,041                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/types.h:3,
2021-04-25T17:28:09,041                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader_options.h:4,
2021-04-25T17:28:09,041                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/base.h:3,
2021-04-25T17:28:09,041                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/stateful.h:3,
2021-04-25T17:28:09,041                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader.h:3,
2021-04-25T17:28:09,041                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data.h:3,
2021-04-25T17:28:09,041                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/all.h:8,
2021-04-25T17:28:09,041                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/extension.h:4,
2021-04-25T17:28:09,041                      from /tmp/pip-req-build-8zq6h56v/csrc/layer_norm_cuda.cpp:1:
2021-04-25T17:28:09,041     /tmp/pip-req-build-8zq6h56v/csrc/layer_norm_cuda.cpp: In function ‘std::vector<at::Tensor> layer_norm_affine(at::Tensor, c10::IntArrayRef, at::Tensor, at::Tensor, double)’:
2021-04-25T17:28:09,041     /tmp/pip-req-build-8zq6h56v/csrc/layer_norm_cuda.cpp:117:42: warning: ‘at::DeprecatedTypeProperties& at::Tensor::type() const’ is deprecated: Tensor.type() is deprecated. Instead use Tensor.options(), which in many cases (e.g. in a constructor) is a drop-in replacement. If you were using data from type(), that is now available from Tensor itself, so instead of tensor.type().scalar_type(), use tensor.scalar_type() instead and instead of tensor.type().backend() use tensor.device(). [-Wdeprecated-declarations]
2021-04-25T17:28:09,041      #define CHECK_CUDA(x) TORCH_CHECK(x.type().is_cuda(), #x " must be a CUDA tensor")
2021-04-25T17:28:09,042                                               ^
2021-04-25T17:28:09,042     /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/c10/macros/Macros.h:173:65: note: in definition of macro ‘C10_UNLIKELY’
2021-04-25T17:28:09,042      #define C10_UNLIKELY(expr)  (__builtin_expect(static_cast<bool>(expr), 0))
2021-04-25T17:28:09,042                                                                      ^~~~
2021-04-25T17:28:09,042     /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/c10/util/Exception.h:362:7: note: in expansion of macro ‘C10_UNLIKELY_OR_CONST’
2021-04-25T17:28:09,042        if (C10_UNLIKELY_OR_CONST(!(cond))) {                                 \
2021-04-25T17:28:09,042            ^~~~~~~~~~~~~~~~~~~~~
2021-04-25T17:28:09,042     /tmp/pip-req-build-8zq6h56v/csrc/layer_norm_cuda.cpp:117:23: note: in expansion of macro ‘TORCH_CHECK’
2021-04-25T17:28:09,042      #define CHECK_CUDA(x) TORCH_CHECK(x.type().is_cuda(), #x " must be a CUDA tensor")
2021-04-25T17:28:09,042                            ^~~~~~~~~~~
2021-04-25T17:28:09,042     /tmp/pip-req-build-8zq6h56v/csrc/layer_norm_cuda.cpp:119:24: note: in expansion of macro ‘CHECK_CUDA’
2021-04-25T17:28:09,042      #define CHECK_INPUT(x) CHECK_CUDA(x); CHECK_CONTIGUOUS(x)
2021-04-25T17:28:09,042                             ^~~~~~~~~~
2021-04-25T17:28:09,042     /tmp/pip-req-build-8zq6h56v/csrc/layer_norm_cuda.cpp:149:3: note: in expansion of macro ‘CHECK_INPUT’
2021-04-25T17:28:09,042        CHECK_INPUT(input);
2021-04-25T17:28:09,042        ^
2021-04-25T17:28:09,042     In file included from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/ATen/Tensor.h:3:0,
2021-04-25T17:28:09,043                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/ATen/Context.h:4,
2021-04-25T17:28:09,043                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/ATen/ATen.h:9,
2021-04-25T17:28:09,043                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/types.h:3,
2021-04-25T17:28:09,043                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader_options.h:4,
2021-04-25T17:28:09,043                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/base.h:3,
2021-04-25T17:28:09,043                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/stateful.h:3,
2021-04-25T17:28:09,043                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader.h:3,
2021-04-25T17:28:09,043                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data.h:3,
2021-04-25T17:28:09,043                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/all.h:8,
2021-04-25T17:28:09,043                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/extension.h:4,
2021-04-25T17:28:09,043                      from /tmp/pip-req-build-8zq6h56v/csrc/layer_norm_cuda.cpp:1:
2021-04-25T17:28:09,043     /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/ATen/core/TensorBody.h:303:30: note: declared here
2021-04-25T17:28:09,043        DeprecatedTypeProperties & type() const {
2021-04-25T17:28:09,043                                   ^~~~
2021-04-25T17:28:09,043     In file included from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/c10/core/DeviceType.h:8:0,
2021-04-25T17:28:09,043                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/c10/core/Device.h:3,
2021-04-25T17:28:09,044                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/c10/core/Allocator.h:6,
2021-04-25T17:28:09,044                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/ATen/ATen.h:7,
2021-04-25T17:28:09,044                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/types.h:3,
2021-04-25T17:28:09,044                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader_options.h:4,
2021-04-25T17:28:09,044                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/base.h:3,
2021-04-25T17:28:09,044                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/stateful.h:3,
2021-04-25T17:28:09,044                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader.h:3,
2021-04-25T17:28:09,044                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data.h:3,
2021-04-25T17:28:09,044                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/all.h:8,
2021-04-25T17:28:09,044                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/extension.h:4,
2021-04-25T17:28:09,044                      from /tmp/pip-req-build-8zq6h56v/csrc/layer_norm_cuda.cpp:1:
2021-04-25T17:28:09,044     /tmp/pip-req-build-8zq6h56v/csrc/layer_norm_cuda.cpp:117:42: warning: ‘at::DeprecatedTypeProperties& at::Tensor::type() const’ is deprecated: Tensor.type() is deprecated. Instead use Tensor.options(), which in many cases (e.g. in a constructor) is a drop-in replacement. If you were using data from type(), that is now available from Tensor itself, so instead of tensor.type().scalar_type(), use tensor.scalar_type() instead and instead of tensor.type().backend() use tensor.device(). [-Wdeprecated-declarations]
2021-04-25T17:28:09,044      #define CHECK_CUDA(x) TORCH_CHECK(x.type().is_cuda(), #x " must be a CUDA tensor")
2021-04-25T17:28:09,044                                               ^
2021-04-25T17:28:09,044     /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/c10/macros/Macros.h:173:65: note: in definition of macro ‘C10_UNLIKELY’
2021-04-25T17:28:09,044      #define C10_UNLIKELY(expr)  (__builtin_expect(static_cast<bool>(expr), 0))
2021-04-25T17:28:09,045                                                                      ^~~~
2021-04-25T17:28:09,045     /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/c10/util/Exception.h:362:7: note: in expansion of macro ‘C10_UNLIKELY_OR_CONST’
2021-04-25T17:28:09,045        if (C10_UNLIKELY_OR_CONST(!(cond))) {                                 \
2021-04-25T17:28:09,045            ^~~~~~~~~~~~~~~~~~~~~
2021-04-25T17:28:09,045     /tmp/pip-req-build-8zq6h56v/csrc/layer_norm_cuda.cpp:117:23: note: in expansion of macro ‘TORCH_CHECK’
2021-04-25T17:28:09,045      #define CHECK_CUDA(x) TORCH_CHECK(x.type().is_cuda(), #x " must be a CUDA tensor")
2021-04-25T17:28:09,045                            ^~~~~~~~~~~
2021-04-25T17:28:09,045     /tmp/pip-req-build-8zq6h56v/csrc/layer_norm_cuda.cpp:119:24: note: in expansion of macro ‘CHECK_CUDA’
2021-04-25T17:28:09,045      #define CHECK_INPUT(x) CHECK_CUDA(x); CHECK_CONTIGUOUS(x)
2021-04-25T17:28:09,045                             ^~~~~~~~~~
2021-04-25T17:28:09,045     /tmp/pip-req-build-8zq6h56v/csrc/layer_norm_cuda.cpp:150:3: note: in expansion of macro ‘CHECK_INPUT’
2021-04-25T17:28:09,045        CHECK_INPUT(gamma);
2021-04-25T17:28:09,045        ^
2021-04-25T17:28:09,045     In file included from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/ATen/Tensor.h:3:0,
2021-04-25T17:28:09,045                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/ATen/Context.h:4,
2021-04-25T17:28:09,045                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/ATen/ATen.h:9,
2021-04-25T17:28:09,046                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/types.h:3,
2021-04-25T17:28:09,046                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader_options.h:4,
2021-04-25T17:28:09,046                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/base.h:3,
2021-04-25T17:28:09,046                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/stateful.h:3,
2021-04-25T17:28:09,046                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader.h:3,
2021-04-25T17:28:09,046                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data.h:3,
2021-04-25T17:28:09,046                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/all.h:8,
2021-04-25T17:28:09,046                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/extension.h:4,
2021-04-25T17:28:09,046                      from /tmp/pip-req-build-8zq6h56v/csrc/layer_norm_cuda.cpp:1:
2021-04-25T17:28:09,046     /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/ATen/core/TensorBody.h:303:30: note: declared here
2021-04-25T17:28:09,046        DeprecatedTypeProperties & type() const {
2021-04-25T17:28:09,046                                   ^~~~
2021-04-25T17:28:09,046     In file included from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/c10/core/DeviceType.h:8:0,
2021-04-25T17:28:09,046                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/c10/core/Device.h:3,
2021-04-25T17:28:09,046                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/c10/core/Allocator.h:6,
2021-04-25T17:28:09,047                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/ATen/ATen.h:7,
2021-04-25T17:28:09,047                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/types.h:3,
2021-04-25T17:28:09,047                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader_options.h:4,
2021-04-25T17:28:09,047                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/base.h:3,
2021-04-25T17:28:09,047                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/stateful.h:3,
2021-04-25T17:28:09,047                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader.h:3,
2021-04-25T17:28:09,047                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data.h:3,
2021-04-25T17:28:09,047                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/all.h:8,
2021-04-25T17:28:09,047                      from /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/torch/extension.h:4,
2021-04-25T17:28:09,047                      from /tmp/pip-req-build-8zq6h56v/csrc/layer_norm_cuda.cpp:1:
2021-04-25T17:28:09,047     /tmp/pip-req-build-8zq6h56v/csrc/layer_norm_cuda.cpp:117:42: warning: ‘at::DeprecatedTypeProperties& at::Tensor::type() const’ is deprecated: Tensor.type() is deprecated. Instead use Tensor.options(), which in many cases (e.g. in a constructor) is a drop-in replacement. If you were using data from type(), that is now available from Tensor itself, so instead of tensor.type().scalar_type(), use tensor.scalar_type() instead and instead of tensor.type().backend() use tensor.device(). [-Wdeprecated-declarations]
2021-04-25T17:28:09,047      #define CHECK_CUDA(x) TORCH_CHECK(x.type().is_cuda(), #x " must be a CUDA tensor")
2021-04-25T17:28:09,047                                               ^
2021-04-25T17:28:09,047     /home/partha9/py38_apex/lib/python3.8/site-packages/torch/include/c10/macros/Macros.h:173:65: note: in definitio

Start with setup.py and csrc/flatten_unflatten.cpp, then inspect the full build log around the cublasLtMatmulDescOpaque_t error and the reported PyTorch 1.8.1/CUDA 10.2 environment. Done means identifying whether this version combination is supported and documenting or validating a reproducible resolution for the Apex build.