[xccl] Flight recorder does not mark completed works: state stays "scheduled"/"started" and last_completed_collective is -1
- Dominant language
- Python
- Stars
- 113
- Forks
- 128
- Avg merge
- 5d 9h
- Merged PRs (30d)
- 112
Description
> Drafted with assistance from an AI coding agent (Claude Code) and reviewed/verified by me on 4x Intel Data Center GPU Max 1550.
### Summary
XCCL's flight recorder never transitions entries to the `completed` state, so
trace-verification assertions that pass on NCCL fail on XPU.
### Observed vs expected
Running the upstream flight-recorder tests against XCCL (ported in place to be
device-agnostic), three fields disagree with the NCCL behaviour:
| Field | XCCL | NCCL |
|---|---|---|
| `pg_status["0"]["last_completed_collective"]` | `-1` | `2` |
| `entries[-1]["state"]` | `scheduled` / `started` | `completed` |
| `entries[-1]["time_discovered_completed_ns"]` | `None` | not `None` |
Everything else in the trace is correct: `pg_config`, `ranks`, `record_id`,
`thread_id`/`thread_name`, entry counts, and `profiling_name` (modulo the
expected `xccl:` vs `nccl:` prefix).
### Reproduction
PyTorch `2.15.0a0+git4d3ac9e`, oneCCL 2022.1, 4x Intel Data Center GPU Max 1550,
`triton-xpu 3.8.0+git1e2d42a0`.
The equivalent check in torch-xpu-ops' own
`test/xpu/distributed/test_c10d_xccl.py` (branch `daisyden/distributed_2.13`) is
commented out with the note "Watchdog will fix marking works completed", which
suggests this is known:
# self.assertEqual(str(pg_status["0"]["last_completed_collective"]), "2")
# self.assertEqual(last["state"], "completed") # Watchdog will fix marking works completed
### Impact
8 test methods (18 parametrizations) in `test/distributed/test_c10d_nccl.py`
`NCCLTraceTest` cannot be enabled for XPU: `test_short_json`,
`test_short_pickle`, `test_long`, `test_trace_while_all_works_retired`,
`test_trace_while_active`, `test_batched_send_recv`,
`test_batched_send_recv_compiled`, `test_individual_send_recv`.
**Before closing this ticket, remove the corresponding `@skipIfXpu` lines from the upstream tests.**
Contributor guide
Research direction
Start by running the NCCLTraceTest cases in test/distributed/test_c10d_nccl.py and compare their XPU behavior with the reported fields. Trace the XCCL flight-recorder completion path, including the watchdog mentioned in the issue; done means completed states, completion timestamps, and last_completed_collective match expectations, then remove the corresponding @skipIfXpu lines.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 58/100