nvidia build issues

o, I have now 4 solid test scenarios thanks to everyone's help here. The have all been tested in cpu mode. I am now switching to nvidia and the docker container doesn't seem to build.

I will be testing resnet, retina, bert and gpt
command to set up container:

cm run script --tags=run-mlperf,inference,_find-performance,_full,_r4.1 \
   --model=resnet50 \
   --implementation=nvidia \
   --framework=tensorrt \
   --category=edge \
   --scenario=Offline \
   --execution_mode=test \
   --device=cuda  \
   --docker --quiet \
   --test_query_count=1000

error:

INFO:root:ENV[CM_CUDA_PATH_LIB_CUDNN_EXISTS]: no
INFO:root:ENV[CM_CUDA_VERSION]: 11.8
INFO:root:ENV[CM_CUDA_VERSION_STRING]: cu118
INFO:root:ENV[CM_NVCC_BIN_WITH_PATH]: /home/fuseadm/CM/repos/local/cache/af058f4059c24b16/install/bin/nvcc
INFO:root:ENV[CUDA_HOME]: /home/fuseadm/CM/repos/local/cache/af058f4059c24b16/install
INFO:root:           ! cd /home/fuseadm
INFO:root:           ! call /home/fuseadm/CM/repos/mlcommons@cm4mlops/script/get-cuda-devices/run.sh from tmp-run.sh
rm: cannot remove 'a.out': No such file or directory

Checking compiler version ...

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Wed_Sep_21_10:33:58_PDT_2022
Cuda compilation tools, release 11.8, V11.8.89
Build cuda_11.8.r11.8/compiler.31833905_0

Compiling program ...


Running program ...

INFO:root:========================================================
INFO:root:Print file tmp-run.out:
INFO:root:
INFO:root:Error: problem obtaining number of CUDA devices: 35

INFO:root:

CM error: Portable CM script failed (name = get-cuda-devices, return code = 256)

Also is there a simple way to reset / clear back to step 1?

Reproduce the exact cm run script command and inspect the get-cuda-devices/run.sh step that reports CUDA error 35. Check whether the container can detect the NVIDIA devices, then confirm that the requested MLPerf scenarios build and run successfully in NVIDIA mode.