intel / intel/llvm

[SYCL][XPTI] Several issues in XPTI

Open
#7,860 3 comments 0 reactions 1 assignee Claimed by @tovinkere View on GitHub
bug confirmed
Dominant language
LLVM
Stars
1.5k
Forks
854
Avg merge
3d 17h
Merged PRs (30d)
137

Description

Hello!
I performed some experiments with XPTI and found some issues.

1. MemCpyCommand::emitInstrumentationData() and MemCpyCommandHost::emitInstrumentationData() add copy_from and copy_to metadata as pointer to device, instead values returned by `deviceToID`.
See: https://github.com/intel/llvm/issues/6972

2. Generated unique id (in default XPTI Framework) can be the same for different payloads. It's sporadically reproduced and in most cases it associated with `codeptr` value.
It can be reproduced with https://github.com/intel/pti-gpu/tree/master/samples/dpc_gemm.
Most often reproduced with CUDA backend. Memory Allocation and Memory Deallocation nodes can have the same unique id.
Probably it can be fixed if use payload value as key instead unique id in `MEvents`.

3. For MemCpyCommand also can be generated equal unique id value because it use the same name in payload (Op name + ptr to memory). But operation can have different source and destination.
For example Memory Transfer[0x12345], from device1 to device2 and Memory Transfer[0x12345], from device2 to device1 will have the same unique id value, but it's different node in graph.
Possible fix: add postfix for operation name with src and dst devices id.
Also can be reproduced with https://github.com/intel/pti-gpu/tree/master/samples/dpc_gemm on CUDA backend.

4. Gathering statistic from event to get execution time of kernel.

4.1 `xpti::trace_point_type_t::signal` send to subscriber pointer to `_pi_event` in `user_data`.
But we can't understood how to interpret this value due to different backends have different implementation of `_pi_event`.
We can get all intresting us information via `_pi_plugin*`, but we also haven't it.
So I guess that `xpti::trace_point_type_t::signal` also must sent to ubscriber pointer to `_pi_plugin` for corresponding `_pi_event`.

4.2 `_pi_event` returned by `xpti::trace_point_type_t::signal` will release with queue destroing, but we have no notification about it.
And if tool based on XPTI tried to gather statistic about this event it will get error.
Possible fix: notify about `_pi_event` releasing or kernel completition execution (not sure, but I guess it's not possible). Or release `_pi_event` with destroing DPC++ enviroment (.so unloading).

4.3 `Or release _pi_event with destroing DPC++ enviroment (.so unloading).` - this option doesn't work for CUDA backend.
Looks like some `CUevent` evBase_, evEnd_ or evStart_ invalidated after destroing queue.

4.4 By default queue created without `sycl::property::queue::enable_profiling()` so we can't gather statistics if in user application haven't enabled profiling.
https://github.com/intel/llvm/blob/sycl/sycl/plugins/cuda/pi_cuda.cpp#L3885
I guess in case if `xptiTraceEnabled() == true` we should enable profiling in queue.

This problems appear if run on Intel iGPU (OpenCL backend) and NVIDIA GPU (CUDA backend).

**Enviroment:**
CPU: Intel(R) Core(TM) i5-9300H CPU
GPU: GeForce GTX 1650

DPC++:
commit 4043dda356af59d3d88607037955a0728dc0f466 (tag: 2022-06)
Author: Steffen Larsen
Date: Tue Jun 14 20:26:32 2022 +0100

[SYCL] Bump sycl library version to 5.7.0 (#6303)

Increases minor version of the DPC++ runtime library in preparation for
the next release. The new version is 5.7.0.

Signed-off-by: Larsen, Steffen

sycl-ls output:
[opencl:gpu:0] Intel(R) OpenCL HD Graphics, Intel(R) Graphics [0x3e9b] 3.0 [20.52.18783]
[ext_oneapi_cuda:gpu:0] NVIDIA CUDA BACKEND, GeForce GTX 1650 0.0 [CUDA 11.0]
[host:host:0] SYCL host platform, SYCL host device 1.2 [1.2]

OS: Ubuntu 18.04

@tovinkere take a look please

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.