wrong linkage between GPU ops and their CPU parents
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 196
- Forks
- 84
- PR merge metrics
- No merged PRs in 30d
Description
Introduced in: https://github.com/mlcommons/chakra/pull/185
Previously, linkage was done based on the Ev idx attribute of the nodes and on the timestamps. This PR introduced linkage based on External id attribute, but this fails to properly link the collective communication nodes to their CPU parent.
For example, a ncclDevKernel... operation (which is a collective communication) should be linked to a nccl:.. operation (which is a runtime op, as far as I believe) and nccl:... should be linked to a record_param_comms operation. This was the behavior before the PR.
Now, a ncclDevKernel... is wrongly linked to a record_param_comms, thus basically ending up with the nccl:... runtime operation (which is a COMP_NODE) being executed in parallel with the ncclDevKernel....
The cause of the wrong linkage is the usage of external id, as can be seen in the following example:
ncclDevKernel_AllGather_RING_LL with External id 17
nccl:_all_gather_base with External id 18
record_param_comms with External id 17
The dependencies should be as follows: record_param_comms -> nccl:_all_gather_base ->ncclDevKernel_AllGather_RING_LL, but because External id does not match, the ncclDevKernel_AllGather_RING_LL is linked to record_param_comms instead of nccl:_all_gather_base.
Furthermore, this is confirmed by analyzing the timestamps of each operation: the closest to ncclDevKernel_AllGather_RING_LL in terms of time is nccl:_all_gather_base and not record_param_comms.
@JoongunPark , any particular reason for changing Ev idx linkage to External id? Is Ev idx unreliable?
cc: @winstonliu1111 @danmih-ixia
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing PR #185 and the linkage logic it introduced, then reproduce the example with ncclDevKernel_AllGather_RING_LL, nccl:_all_gather_base, and record_param_comms. Compare External id linkage with the previous Ev idx and timestamp behavior; done means the dependency chain is record_param_comms -> nccl:_all_gather_base -> ncclDevKernel_AllGather_RING_LL.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100