intel / intel/llvm

HIP profiling submission time query returns weird values

Open
#12,904 5 comments 0 reactions 0 assignees View on GitHub
bug hip
Dominant language
LLVM
Stars
1.5k
Forks
854
Avg merge
3d 17h
Merged PRs (30d)
137

Description

### Describe the bug

In https://github.com/intel/llvm/pull/12838 it seems like the submission time on HIP is giving weird values. I did a bit of digging and it seems to me like HIP is a little different from CUDA when checking timing-differences between events. Of particular interest here is the following line for [hipEventElapsedTime()](https://docs.amd.com/projects/HIP/en/docs-5.4.0/doxygen/html/group___event.html#gad4128b815cb475c8e13c7e66ff6250b7):

> Events which are recorded in a NULL stream will block until all commands on all other streams complete execution, and then record the timestamp.

While what we expect here is to get an event with the current time, hence using an otherwise unused stream. Fixing it might be outside the scope of this PR, but a possible solution could be to lazily have a stream specifically for recording submission time of events, tied to the context. Similar could be used in the CUDA backend to avoid the assumption noted above.

_Originally posted by @steffenlarsen in https://github.com/oneapi-src/unified-runtime/pull/1400#discussion_r1508693119_

### To reproduce

_No response_

### Environment

_No response_

### Additional context

This affects https://github.com/intel/llvm/pull/12838, but should be reproducible on normal profiling queues. When this is fixed, sycl/test-e2e/ProfilingTag/ tests should be enabled for HIP.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.