NVIDIA / NVIDIA/TensorRT-LLM

run nccl_test error

Open
#3,480 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug Scale-out Testing
Dominant language
Python
Stars
14.7k
Forks
2.8k
Avg merge
2d 23h
Merged PRs (30d)
489

Description

System Info

cpu: arm64
gpu: Orin
error message: NCCL WARN Cuda failure 'named symbol not found'

Who can help?

No response

Information
  • The official example scripts
  • My own modified scripts
Tasks
  • An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
  • My own task or dataset (give details below)
Reproduction

download nccl_test :git clone https://github.com/NVIDIA/nccl-tests.git && cd nccl-tests
compile: make MPI=0
run example and then the error message is coming .

Expected behavior

run nccl_test example ok.

actual behavior

run nccl_test example error

additional notes

no

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the failure by cloning nccl-tests, entering that checkout, and running make MPI=0 before the example command. Compare the Orin arm64 environment and reported NCCL WARN Cuda failure 'named symbol not found' with the expected successful run; done means the nccl_test example runs without the error.

Written by the indexing model from the issue text.

Assessment

Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.