Kineto collection of nccl:coalesced don't record metadata
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 196
- Forks
- 84
- PR merge metrics
- No merged PRs in 30d
Description
Chakra's kineto trace add metadata to the NCCL kernels, including message size, PG attributes (name, description, ranks, etc).
However in a trace I just recorded some kernels don't include any metadata.
The trace records gpt3/175b_fp8 training script from NeMo Launcher.
Here is the report of a normal all_reduce operation, we can see info about PG.
While here is the report of the nccl coalesced allgather, which displays no metadata:
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the Chakra kineto trace from the gpt3/175b_fp8 training script and compare a normal all_reduce with the nccl coalesced allgather. Trace the metadata collection path for both operations; done means coalesced NCCL kernels include the expected message size and process-group metadata.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, observability-sre
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100