Dist DGL not working with macOS Monterey
- Dominant language
- Python
- Stars
- 14.3k
- Forks
- 3.1k
- PR merge metrics
- No merged PRs in 30d
Description
## 🐛 Bug
When I am running ```dgl``` distributedly with macOS Monterey I am receiving an error like this ```NameError: name '_CAPI_DGLRPCHandleSignal' is not defined```
Discussion about this can also be found [here](https://discuss.dgl.ai/t/distributed-training-not-working-on-two-apple-imac-devices/2544)
## To Reproduce
Steps to reproduce the behavior:
1. Follow the tutorial [here](https://docs.dgl.ai/en/latest/tutorials/dist/1_node_classification.html)
2. Run it on macOS Monterey
```
Traceback (most recent call last):
File "train_dist.py", line 13, in
dgl.distributed.initialize(ip_config='ip_config.txt')
File "/Users/ccsp-admin/opt/anaconda3/envs/dist-gnn/lib/python3.7/site-packages/dgl/distributed/dist_context.py", line 261, in initialize
connect_to_server(ip_config, num_servers, max_queue_size, net_type)
File "/Users/ccsp-admin/opt/anaconda3/envs/dist-gnn/lib/python3.7/site-packages/dgl/distributed/rpc_client.py", line 141, in connect_to_server
rpc.register_sig_handler()
File "/Users/ccsp-admin/opt/anaconda3/envs/dist-gnn/lib/python3.7/site-packages/dgl/distributed/rpc.py", line 992, in register_sig_handler
_CAPI_DGLRPCHandleSignal()
NameError: name '_CAPI_DGLRPCHandleSignal' is not defined
Traceback (most recent call last):
File "train_dist.py", line 14, in
dgl.distributed.initialize(ip_config='ip_config.txt')
File "/Users/ccsp-admin/opt/anaconda3/envs/dist-gnn/lib/python3.7/site-packages/dgl/distributed/dist_context.py", line 261, in initialize
connect_to_server(ip_config, num_servers, max_queue_size, net_type)
File "/Users/ccsp-admin/opt/anaconda3/envs/dist-gnn/lib/python3.7/site-packages/dgl/distributed/rpc_client.py", line 141, in connect_to_server
rpc.register_sig_handler()
File "/Users/ccsp-admin/opt/anaconda3/envs/dist-gnn/lib/python3.7/site-packages/dgl/distributed/rpc.py", line 992, in register_sig_handler
_CAPI_DGLRPCHandleSignal()
NameError: name '_CAPI_DGLRPCHandleSignal' is not defined
libc++abi: terminating with uncaught exception of type dmlc::Error: [16:43:32] /tmp/dgl_src/src/runtime/shared_mem.cc:55: Check failed: munmap(ptr_, size_) != -1: Invalid argument
Stack trace:
[bt] (0) 1 libdgl.dylib 0x000000013755a14f dmlc::LogMessageFatal::~LogMessageFatal() + 111
[bt] (1) 2 libdgl.dylib 0x0000000137f2f516 dgl::runtime::SharedMemory::~SharedMemory() + 150
[bt] (2) 3 libdgl.dylib 0x0000000137f738c0 dgl::HeteroGraph::CopyToSharedMem(std::__1::shared_ptr, std::__1::basic_string, std::__1::allocator > const&, std::__1::vector, std::__1::allocator >, std::__1::allocator, std::__1::allocator > > > const&, std::__1::vector, std::__1::allocator >, std::__1::allocator, std::__1::allocator > > > const&, std::__1::set, std::__1::allocator >, std::__1::less, std::__1::allocator > >, std::__1::allocator, std::__1::allocator > > > const&) + 7104
[bt] (3) 4 libdgl.dylib 0x0000000137f8ab15 std::__1::__function::__func, void (dgl::runtime::DGLArgs, dgl::runtime::DGLRetValue*)>::operator()(dgl::runtime::DGLArgs&&, dgl::runtime::DGLRetValue*&&) + 677
[bt] (4) 5 libdgl.dylib 0x0000000137f12fb8 DGLFuncCall + 72
[bt] (5) 6 core.cpython-37m-darwin.so 0x000000010eea01a5 __pyx_f_3dgl_4_ffi_4_cy3_4core_FuncCall(void*, _object*, DGLValue*, int*) + 965
[bt] (6) 7 core.cpython-37m-darwin.so 0x000000010eea43f4 __pyx_pw_3dgl_4_ffi_4_cy3_4core_12FunctionBase_5__call__(_object*, _object*, _object*) + 52
[bt] (7) 8 python3.7 0x0000000104071bbb _PyObject_FastCallKeywords + 683
[bt] (8) 9 python3.7 0x000000010417a565 call_function + 725
Exception in thread Thread-2:
Traceback (most recent call last):
File "/Users/ccsp-admin/opt/anaconda3/envs/dist-gnn/lib/python3.7/threading.py", line 926, in _bootstrap_inner
self.run()
File "/Users/ccsp-admin/opt/anaconda3/envs/dist-gnn/lib/python3.7/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "/Users/ccsp-admin/workspace/dgl/tools/launch.py", line 112, in run
subprocess.check_call(ssh_cmd, shell=True)
File "/Users/ccsp-admin/opt/anaconda3/envs/dist-gnn/lib/python3.7/subprocess.py", line 363, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'ssh -o StrictHostKeyChecking=no -p 22 192.168.1.98 'cd /Users/ccsp-admin/workspace/; . ~/.zshrc; conda activate dist-gnn; (export DGL_ROLE=server DGL_NUM_SAMPLER=0 OMP_NUM_THREADS=1 DGL_NUM_CLIENT=2 DGL_CONF_PATH=2part_data/ogbn-proteins.json DGL_IP_CONFIG=ip_config.txt DGL_NUM_SERVER=1 DGL_GRAPH_FORMAT=csc DGL_SERVER_ID=1; python3 train_dist.py)'' returned non-zero exit status 255.
libc++abi: terminating with uncaught exception of type dmlc::Error: [16:43:33] /tmp/dgl_src/src/runtime/shared_mem.cc:55: Check failed: munmap(ptr_, size_) != -1: Invalid argument
Stack trace:
[bt] (0) 1 libdgl.dylib 0x000000016b37714f dmlc::LogMessageFatal::~LogMessageFatal() + 111
[bt] (1) 2 libdgl.dylib 0x000000016bd4c516 dgl::runtime::SharedMemory::~SharedMemory() + 150
[bt] (2) 3 libdgl.dylib 0x000000016bd908c0 dgl::HeteroGraph::CopyToSharedMem(std::__1::shared_ptr, std::__1::basic_string, std::__1::allocator > const&, std::__1::vector, std::__1::allocator >, std::__1::allocator, std::__1::allocator > > > const&, std::__1::vector, std::__1::allocator >, std::__1::allocator, std::__1::allocator > > > const&, std::__1::set, std::__1::allocator >, std::__1::less, std::__1::allocator > >, std::__1::allocator, std::__1::allocator > > > const&) + 7104
[bt] (3) 4 libdgl.dylib 0x000000016bda7b15 std::__1::__function::__func, void (dgl::runtime::DGLArgs, dgl::runtime::DGLRetValue*)>::operator()(dgl::runtime::DGLArgs&&, dgl::runtime::DGLRetValue*&&) + 677
[bt] (4) 5 libdgl.dylib 0x000000016bd2ffb8 DGLFuncCall + 72
[bt] (5) 6 core.cpython-37m-darwin.so 0x000000012bc591a5 __pyx_f_3dgl_4_ffi_4_cy3_4core_FuncCall(void*, _object*, DGLValue*, int*) + 965
[bt] (6) 7 core.cpython-37m-darwin.so 0x000000012bc5d3f4 __pyx_pw_3dgl_4_ffi_4_cy3_4core_12FunctionBase_5__call__(_object*, _object*, _object*) + 52
[bt] (7) 8 python3.7 0x00000001021f4bbb _PyObject_FastCallKeywords + 683
[bt] (8) 9 python3.7 0x00000001022fd565 call_function + 725
Exception in thread Thread-1:
Traceback (most recent call last):
File "/Users/ccsp-admin/opt/anaconda3/envs/dist-gnn/lib/python3.7/threading.py", line 926, in _bootstrap_inner
self.run()
File "/Users/ccsp-admin/opt/anaconda3/envs/dist-gnn/lib/python3.7/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "/Users/ccsp-admin/workspace/dgl/tools/launch.py", line 112, in run
subprocess.check_call(ssh_cmd, shell=True)
File "/Users/ccsp-admin/opt/anaconda3/envs/dist-gnn/lib/python3.7/subprocess.py", line 363, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'ssh -o StrictHostKeyChecking=no -p 22 192.168.1.237 'cd /Users/ccsp-admin/workspace/; . ~/.zshrc; conda activate dist-gnn; (export DGL_ROLE=server DGL_NUM_SAMPLER=0 OMP_NUM_THREADS=1 DGL_NUM_CLIENT=2 DGL_CONF_PATH=2part_data/ogbn-proteins.json DGL_IP_CONFIG=ip_config.txt DGL_NUM_SERVER=1 DGL_GRAPH_FORMAT=csc DGL_SERVER_ID=0; python3 train_dist.py)'' returned non-zero exit status 255.
```
## Expected behavior
Dist DGl should be working with macOS Monterey
## Environment
- DGL Version (e.g., 1.0): 0.7.2
- Backend Library & Version (e.g., PyTorch 0.4.1, MXNet/Gluon 1.3): Pytorch 1.10.0
- OS (e.g., Linux): macOS Monterey
- How you installed DGL (`conda`, `pip`, source): conda
- Build command you used (if compiling from source):
- Python version: 3.8
- CUDA/cuDNN version (if applicable):
- GPU models and configuration (e.g. V100):
- Any other relevant information:
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.