E3SM-Project / E3SM-Project/EKAT

Remove assumption of CudaUVM in order to create Host team policies

Open
#227 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
C++
Stars
22
Forks
12
PR merge metrics
No merged PRs in 30d

Description

PR #218 added the ability to dispatch a kernel on host when CudaUVM is used, by creating a team policy on host. However, CudaUVM is still a device memory space, with behind-the-scenes sync between host and device. In particular, this means that a View with memory space `HostSpace` cannot be assigned to a view with memory space `CudaUVM`, and viceversa. This can have bad consequences for codes that use host mirrors, and are templated on the memory space. For instance, the following won't work:

```
template
struct Types {
using view = Kokkos::View;
};

typename Types::view my_view_d ("",10);
typename Types::view my_view_h = Kokkos::create_host_mirror(my_view_d); // ERROR!
```
since the host mirror still has CudaUVM as mem space (b/c uvm is accessible from host), but CudaUVM is not assignable to HostSpace. On the other hand, CudaHostPinnedSpace is both assignable and accessible from device and host. The snippet above would work with `CudaHostPinnedSpace` in place of `CudaUVM`.

However, notice that nothing in the team policy requires to know what memory space the views that will be accessed will be on. In fact, the concept of "view" is orthogonal to that of a team policy. All that the team policy needs to know is the _execution space_ where the kernel will be run on. We should therefore remove the requirement that CudaUVM is used, and allow the user to create a policy on host regardless. It is up to the user to ensure that the views used in the kernel are accessible on host.

The drawback would be that Kokkos does not offer a configuration option to make `CudaHostPinnedSpace` the default mem space for the `Cuda` execution space (while it does so for `CudaUVM`). Therefore, any usage of `ExecSpace::memory_space` would yield something different from the host pinned space.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.