[xccl] Profiler kernel events lack collective metadata annotations
- Dominant language
- Python
- Stars
- 113
- Forks
- 128
- Avg merge
- 5d 13h
- Merged PRs (30d)
- 107
Description
> Drafted with assistance from an AI coding agent (Claude Code) and reviewed/verified by me on 4x Intel Data Center GPU Max 1550.
### Summary
Kineto `kernel` events for XCCL collectives do not carry the collective metadata
that NCCL kernels get, so profiler traces cannot be correlated with collective
operations on XPU.
### Observed
Profiling a `dist.all_reduce` on XCCL with
`ProfilerActivity.CPU + ProfilerActivity.XPU`, the collective kernel is recorded:
name: oneccl_reduce_base
cat : kernel
but its `args` contain only SYCL/Level-Zero plumbing:
['External id', 'appended', 'appended_rel_to_start', 'context', 'correlation',
'device', 'engine_index', 'engine_ordinal', 'kernel_id', 'l0 queue',
'source_file_name', 'source_line_number', 'submitted',
'submitted_rel_to_start', 'sycl queue', 'sycl_enqk_begin',
'sycl_enqk_begin_rel_to_start', 'sycl_invocation_id', 'sycl_node_id',
'sycl_task_begin', 'sycl_task_begin_rel_to_start']
### Expected
NCCL kernel events are annotated with, at minimum:
Collective name, dtype, In msg nelems, Out msg nelems
### Reproduction
PyTorch `2.15.0a0+git4d3ac9e`, oneCCL 2022.1, 4x Intel Data Center GPU Max 1550.
### Impact
`CommTest::test_profiler_nccl_annotations_on_gpu_kernels` (both
`use_python_export` parametrizations) in `test/distributed/test_c10d_nccl.py`
cannot be enabled for XPU. It fails with
`AssertionError: 0 not greater than 0 : Expected at least one GPU kernel with NCCL annotations`.
**Before closing this ticket, remove the corresponding `@skipIfXpu` lines from the upstream tests.**
Contributor guide
Research direction
Start with CommTest::test_profiler_nccl_annotations_on_gpu_kernels in test/distributed/test_c10d_nccl.py and reproduce the XPU all_reduce trace using the stated PyTorch, oneCCL, and hardware setup. Trace how NCCL kernel metadata reaches Kineto kernel events, then make the XCCL events expose Collective name, dtype, and message sizes; done means both python-export parametrizations pass and the @skipIfXpu lines are removed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100