ImportError: /home/miniconda3/envs/BMCook/lib/python3.10/site-packages/bmtrain/nccl/_C.cpython-310-x86_64-linux-gnu.so: undefined symbol: ncclBroadcast ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 17198) of binary: /home/miniconda3/envs/BMCook/bin/python

Hi, I encountered the error described in the title of this issue, while trying to run the gpt-2 example. Here is my command:

export CUDA_VISIBLE_DEVICES=7
torchrun --nnodes=1 --nproc_per_node=1 --rdzv_id=1 --rdzv_backend=c10d --rdzv_endpoint=localhost ./gpt2_test.py \
    --model gpt2-base \
    --save-dir results/gpt2-prune \
    --data-path ... \
    --cook-config configs/gpt2-prune.json \

It seems that this is an error within the package bmtrain, so could you help figure out what happened or how to avoid it? Thanks a lot!

Start with the reported gpt2_test.py command and the ImportError for bmtrain/nccl/_C.cpython-310-x86_64-linux-gnu.so. Reproduce the failure in the stated Python 3.10 environment, then determine whether the package setup is compatible with the requested gpt-2 example. Done means the example runs without the undefined ncclBroadcast symbol error or the required environment constraints are documented.