facebookresearch / facebookresearch/FAMBench

An error is thrown when running run_dlrm_ubench_train_allreduce.sh

Open
#61 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
32
Forks
25
PR merge metrics
No merged PRs in 30d

Description

When running `mpirun --allow-run-as-root -np 8 -N 8 --bind-to none ./run_dlrm_ubench_train_allreduce.sh -c xxxx`, an error is thrown:

`Traceback (most recent call last):
File "dlrm/ubench/dlrm_ubench_comms_driver.py", line 133, in
main()
File "dlrm/ubench/dlrm_ubench_comms_driver.py", line 106, in main
comms_main()
File "/dockerx/FAMbench/11292021/FAMBench/param/train/comms/pt/comms.py", line 1208, in main
collBenchObj.runBench(comms_world_info, commsParams)
File "/dockerx/FAMbench/11292021/FAMBench/param/train/comms/pt/comms.py", line 1161, in runBench
backendObj.benchmark_comms()
File "/dockerx/FAMbench/11292021/FAMBench/param/train/comms/pt/pytorch_dist_backend.py", line 659, in benchmark_comms
self.commsParams.benchTime(index, self.commsParams, self)
File "/dockerx/FAMbench/11292021/FAMBench/param/train/comms/pt/comms.py", line 1128, in benchTime
self.reportBenchTime(
File "/dockerx/FAMbench/11292021/FAMBench/param/train/comms/pt/comms.py", line 853, in reportBenchTime
self.reportBenchTimeColl(commsParams, results, tensorList)
File "/dockerx/FAMbench/11292021/FAMBench/param/train/comms/pt/comms.py", line 860, in reportBenchTimeColl
latencyAcrossRanks = np.array(tensorList)
File "/opt/conda/lib/python3.8/site-packages/torch/_tensor.py", line 723, in __array__
return self.numpy()
TypeError: can't convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.`

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.