NCCL:Broadcast collectives are missing from the converted trace but present in the trace_link
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 196
- Forks
- 84
- PR merge metrics
- No merged PRs in 30d
Description
Describe the Bug
After running a ResNet50 or TinyLlama2 workload on 4 ranks I see that in the Kineto trace at least one nccl:broadcast collective is observed. In the trace_link file the same collective is observed, but in the converted trace the collective is no longer present. Is this a normal behavior or is it an issue on the Chakra Converter side?
I looked in the converter implementation, but I did not observe any pointers that this should be done - dismiss broadcast collectives. Is there something I missed?
Steps to Reproduce
Using the Chakra version from 6 Sept, after the merge of commit #140.
Expected Behavior
See the nccl:broadcast collective in the converted trace.
Screenshots
This is the trace_link file, the broadcast collective is present.
This is the converted trace, in json format, no broadcast collective can be found - search result is at the bottom of the picture.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the issue with a ResNet50 or TinyLlama2 workload on 4 ranks, then compare the Kineto trace, trace_link file, and converted JSON trace. The work is complete when the converted trace preserves the nccl:broadcast collective observed in the source traces.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- devtools
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100