spcl / spcl/dace

Unexpected Segmentation Fault with Persistent AllocationLifetime

Open
#2,133 7 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
593
Forks
163
Avg merge
2d 23h
Merged PRs (30d)
60

Description

**Context**

I am working on a new `CUDACodeGen` backend as part of my Master's Thesis.
One of my tasks is to improve GPU stream handling, which is currently done implicitly by attaching unregistered GPU stream attributes to
nodes. The new approach makes GPU streams explicit (as presented by Yakup Koray Budanaz in a DaCe meeting) by introducing GPU
stream nodes at the SDFG level. I provided a screenshot below for better understanding. Note that GPU_Device-scheduled maps get
the GPU stream assigned via an AccessNode.



**Bug Description**

The GPU stream AccessNode's data descriptor is an array of GPU streams, created like this:
```python
sdfg.add_transient("gpu_streams", shape, dtype=dace.dtypes.gpuStream_t, storage=dace.dtypes.StorageType.Register,
lifetime=dace.dtypes.AllocationLifetime.Persistent)
```
With this setup, accesses to the GPU stream array only need to ensure the array is properly initialized.
Note that the new array is (as other persistent data) expected to be stored in the `state` struct. Also, In both the new and current
`CUDACodeGen` backends, GPU streams are already stored in the `state` struct (`__state->gpu_context->streams`) . By setting the new
`gpu_streams` array name to reference these streams, the initialization and usage work "automatically".

However, in rare cases, using the Persistent allocation lifetime appears to trigger a segmentation fault. I suspect this because I spent 13 hours
inspecting the differences between the code generated by the new and current backends and could not identify any other cause. Furthermore,
the issue disappears when using the default allocation lifetime. My current hypothesis is that DaCe tries to free the new gpu_streams pointer in
the state struct outside of the generated code, while it is also freed during program exit.

Any insights or suggestions on what might be causing this would be greatly appreciated.

**To Reproduce**

Steps to reproduce the behavior:

1. Go to my forked repo: [https://github.com/aydogdub/dace](https://github.com/aydogdub/dace)
2. Switch to the branch `gpu-streams-management`
3. Run the GPU test at `./tests/npbench/misc/scatterubg_self_test.py`
- Note: This test is marked to be skipped in this branch, so re-enable it before running.
4. Observe the segmentation fault.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.