intel / intel/llvm

[SYCL][CUDA] Copy between device and managed/shared memory fails on WSL

Open
#9,632 1 comment 0 reactions 0 assignees View on GitHub
bug cuda
Dominant language
LLVM
Stars
1.5k
Forks
854
Avg merge
3d 17h
Merged PRs (30d)
137

Description

# Describe the bug

This is a **super obscure** error that I bumped into just now. If we can even call it an error...

One of the unit tests of [our project](https://github.com/acts-project/vecmem) tries to copy data between a memory area in managed/shared memory, and another one in device memory. There is a fair amount of layers between our code and the underlying SYCL code doing that, but that's what's happening here:

https://github.com/acts-project/vecmem/blob/main/tests/sycl/test_sycl_jagged_containers.sycl#L428

This code worked well on all platforms that I have tried until today. But today I tried to make it work on a pretty obscure platform. I'm using a hand-built version of the `2022-12` tag of this repository in WSL, with CUDA 11.7.1 installed in WSL as well, and the latest NVIDIA driver installed on Windows itself. In this definitely non-standard setup that test crashes with the following:

```
...
[ RUN ] sycl_jagged_containers_test.set_in_contiguous_kernel
[ OK ] sycl_jagged_containers_test.set_in_contiguous_kernel (5 ms)
[ RUN ] sycl_jagged_containers_test.filter

Thread 1 "vecmem_test_syc" received signal CUDA_EXCEPTION_15, Invalid Managed Memory Access.
vecmem::copy::copy_views_impl (this=0x7fffffffd730, sizes=..., from_view=0x204e01000, cptype=vecmem::copy::type::unknown, to_view=)
at /mnt/c/Users/krasz/ATLAS/vecmem/vecmem/core/include/vecmem/utils/impl/copy.ipp:415
415 do_copy(sizes[i] * sizeof(TYPE1), from_view[i].ptr(), to_view[i].ptr(),
(cuda-gdb) bt
#0 vecmem::copy::copy_views_impl (this=0x7fffffffd730, sizes=..., from_view=0x204e01000, cptype=vecmem::copy::type::unknown, to_view=)
at /mnt/c/Users/krasz/ATLAS/vecmem/vecmem/core/include/vecmem/utils/impl/copy.ipp:415
#1 vecmem::copy::operator() (this=this@entry=0x7fffffffd730, from_view=..., to_view=..., cptype=cptype@entry=vecmem::copy::type::unknown)
at /mnt/c/Users/krasz/ATLAS/vecmem/vecmem/core/include/vecmem/utils/impl/copy.ipp:296
#2 0x0000000000427d33 in vecmem::copy::operator() > >, std::pmr::polymorphic_allocator > (this=this@entry=0x7fffffffd730, from_view=..., to_vec=..., cptype=cptype@entry=vecmem::copy::type::unknown)
at /mnt/c/Users/krasz/ATLAS/vecmem/vecmem/core/include/vecmem/utils/impl/copy.ipp:326
#3 0x00000000004234a1 in sycl_jagged_containers_test_filter_Test::TestBody (this=this@entry=0xc8a970)
at /mnt/c/Users/krasz/ATLAS/vecmem/vecmem/tests/sycl/test_sycl_jagged_containers.sycl:428
#4 0x00007ffff7f86d29 in testing::internal::HandleSehExceptionsInMethodIfSupported (method=, location=0x7ffff7f932f5 "the test body",
object=) at /home/krasznaa/ATLAS/vecmem/build-llvm/_deps/googletest-src/googletest/src/gtest.cc:2607
#5 testing::internal::HandleExceptionsInMethodIfSupported (object=object@entry=0xc8a970, method=, location=0x7ffff7f932f5 "the test body")
at /home/krasznaa/ATLAS/vecmem/build-llvm/_deps/googletest-src/googletest/src/gtest.cc:2643
#6 0x00007ffff7f62380 in testing::Test::Run (this=this@entry=0xc8a970) at /home/krasznaa/ATLAS/vecmem/build-llvm/_deps/googletest-src/googletest/src/gtest.cc:2682
...
```

There was no deep thinking behind setting up the test like this, it was just convenient for technical reasons. And as soon as I stop using shared memory there and switch to using host memory, this error disappears. But since the error only shows up on WSL, I thought it would be interesting to share this find. :wink:

# To Reproduce

Is a bit difficult. 😦 I described my OS / software setup above. In that environment one can just build https://github.com/acts-project/vecmem/tree/v0.25.0 with its tests, and the error shows up. Unfortunately both setting up this build environment, and then building the project in that environment is not absolutely trivial. So I'd only produce a writeup about it on request...

# Environment (please complete the following information)

- OS: Windows 11, WSL2, Ubuntu 20.04 (All at the same time!)
- Target device and vendor: NVIDIA RTX 3050Ti
- DPC++ version:

```
clang version 16.0.0 (https://github.com/intel/llvm.git 6977f1aced3ed6a08573fdbdd4f35a5d719c8d98)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/krasznaa/software/intel/llvm-2022-12/bin
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/9
Selected GCC installation: /usr/lib/gcc/x86_64-linux-gnu/9
Candidate multilib: .;@m64
Selected multilib: .;@m64
Found CUDA installation: /home/krasznaa/software/cuda/11.7.1, version 11.7
```

- Dependencies version: NVIDIA driver 531.61

Pinging @ivorobts.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.