intel / intel/torch-xpu-ops

[xccl] Flight recorder does not mark completed works: state stays "scheduled"/"started" and last_completed_collective is -1

Open
#5,381 0 comments 0 reactions 0 assignees View on GitHub
module: distributed
Dominant language
Python
Stars
113
Forks
128
Avg merge
5d 9h
Merged PRs (30d)
112

Description

> Drafted with assistance from an AI coding agent (Claude Code) and reviewed/verified by me on 4x Intel Data Center GPU Max 1550.

### Summary

XCCL's flight recorder never transitions entries to the `completed` state, so
trace-verification assertions that pass on NCCL fail on XPU.

### Observed vs expected

Running the upstream flight-recorder tests against XCCL (ported in place to be
device-agnostic), three fields disagree with the NCCL behaviour:

| Field | XCCL | NCCL |
|---|---|---|
| `pg_status["0"]["last_completed_collective"]` | `-1` | `2` |
| `entries[-1]["state"]` | `scheduled` / `started` | `completed` |
| `entries[-1]["time_discovered_completed_ns"]` | `None` | not `None` |

Everything else in the trace is correct: `pg_config`, `ranks`, `record_id`,
`thread_id`/`thread_name`, entry counts, and `profiling_name` (modulo the
expected `xccl:` vs `nccl:` prefix).

### Reproduction

PyTorch `2.15.0a0+git4d3ac9e`, oneCCL 2022.1, 4x Intel Data Center GPU Max 1550,
`triton-xpu 3.8.0+git1e2d42a0`.

The equivalent check in torch-xpu-ops' own
`test/xpu/distributed/test_c10d_xccl.py` (branch `daisyden/distributed_2.13`) is
commented out with the note "Watchdog will fix marking works completed", which
suggests this is known:

# self.assertEqual(str(pg_status["0"]["last_completed_collective"]), "2")
# self.assertEqual(last["state"], "completed") # Watchdog will fix marking works completed

### Impact

8 test methods (18 parametrizations) in `test/distributed/test_c10d_nccl.py`
`NCCLTraceTest` cannot be enabled for XPU: `test_short_json`,
`test_short_pickle`, `test_long`, `test_trace_while_all_works_retired`,
`test_trace_while_active`, `test_batched_send_recv`,
`test_batched_send_recv_compiled`, `test_individual_send_recv`.

**Before closing this ticket, remove the corresponding `@skipIfXpu` lines from the upstream tests.**

Contributor guide

Open the contributing guide

Research direction

Start by running the NCCLTraceTest cases in test/distributed/test_c10d_nccl.py and compare their XPU behavior with the reported fields. Trace the XCCL flight-recorder completion path, including the watchdog mentioned in the issue; done means completed states, completion timestamps, and last_completed_collective match expectations, then remove the corresponding @skipIfXpu lines.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.