intel / intel/torch-xpu-ops

[xccl] Profiler kernel events lack collective metadata annotations

Open
#5,384 0 comments 0 reactions 0 assignees View on GitHub
module: distributed
Dominant language
Python
Stars
113
Forks
128
Avg merge
5d 13h
Merged PRs (30d)
107

Description

> Drafted with assistance from an AI coding agent (Claude Code) and reviewed/verified by me on 4x Intel Data Center GPU Max 1550.

### Summary

Kineto `kernel` events for XCCL collectives do not carry the collective metadata
that NCCL kernels get, so profiler traces cannot be correlated with collective
operations on XPU.

### Observed

Profiling a `dist.all_reduce` on XCCL with
`ProfilerActivity.CPU + ProfilerActivity.XPU`, the collective kernel is recorded:

name: oneccl_reduce_base
cat : kernel

but its `args` contain only SYCL/Level-Zero plumbing:

['External id', 'appended', 'appended_rel_to_start', 'context', 'correlation',
'device', 'engine_index', 'engine_ordinal', 'kernel_id', 'l0 queue',
'source_file_name', 'source_line_number', 'submitted',
'submitted_rel_to_start', 'sycl queue', 'sycl_enqk_begin',
'sycl_enqk_begin_rel_to_start', 'sycl_invocation_id', 'sycl_node_id',
'sycl_task_begin', 'sycl_task_begin_rel_to_start']

### Expected

NCCL kernel events are annotated with, at minimum:

Collective name, dtype, In msg nelems, Out msg nelems

### Reproduction

PyTorch `2.15.0a0+git4d3ac9e`, oneCCL 2022.1, 4x Intel Data Center GPU Max 1550.

### Impact

`CommTest::test_profiler_nccl_annotations_on_gpu_kernels` (both
`use_python_export` parametrizations) in `test/distributed/test_c10d_nccl.py`
cannot be enabled for XPU. It fails with
`AssertionError: 0 not greater than 0 : Expected at least one GPU kernel with NCCL annotations`.

**Before closing this ticket, remove the corresponding `@skipIfXpu` lines from the upstream tests.**

Contributor guide

Open the contributing guide

Research direction

Start with CommTest::test_profiler_nccl_annotations_on_gpu_kernels in test/distributed/test_c10d_nccl.py and reproduce the XPU all_reduce trace using the stated PyTorch, oneCCL, and hardware setup. Trace how NCCL kernel metadata reaches Kineto kernel events, then make the XCCL events expose Collective name, dtype, and message sizes; done means both python-export parametrizations pass and the @skipIfXpu lines are removed.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.