AdaptiveCpp / AdaptiveCpp/AdaptiveCpp

allocation_map lifetime issues with ACPP_PERSISTENT_RUNTIME

Open
#2,208 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
C++
Stars
1.9k
Forks
228
Avg merge
4d 1h
Merged PRs (30d)
25

Description

**Bug summary**

When `ACPP_PERSISTENT_RUNTIME=1` is set, the application sporadically segfaults at shutdown when a backend with allocation tracking is used (verified with OpenCL and Vulkan).

**To Reproduce**

The example below most reliably triggers the failure. I've seen it even with a single `foo` invocation, but is takes longer to manifest.

```console
$ acpp sycl-app-reinit.cpp --acpp-targets=generic -O3 -g
$ i=0; while true; do i=$(($i+1)); ACPP_PERSISTENT_RUNTIME=1 ACPP_VISIBILITY_MASK=opencl:A770 ./a.out &>/dev/null || break; done; echo "Crashed after $i repeats"
Aborted (core dumped)
Crashed after 17 repeats
$ i=0; while true; do i=$(($i+1)); ACPP_PERSISTENT_RUNTIME=1 ACPP_VISIBILITY_MASK=vk:A770 ./a.out &>/dev/null || break; done; echo "Crashed after $i repeats"
Segmentation fault (core dumped)
Crashed after 6 repeats
$ i=0; while true; do i=$(($i+1)); ACPP_PERSISTENT_RUNTIME=1 ACPP_VISIBILITY_MASK=vk:NVIDIA ./a.out &>/dev/null || break; done; echo "Crashed after $i repeats"
Segmentation fault (core dumped)
Crashed after 4 repeats
```

sycl-app-reinit.cpp

```cpp
#include

#include

class Kernel;

void foo() {
// Creating buffer of 4 elements to be used inside the kernel code
sycl::buffer Buffer(4);

for (const auto &Device : sycl::device::get_devices()) {
// Creating SYCL queue
//sycl::queue queue(Device, {sycl::property::queue::in_order()});
sycl::queue queue{Device, {sycl::property::queue::AdaptiveCpp_priority{1}, sycl::property::queue::in_order()}};

std::cout << "Running on "
<< queue.get_device().get_info()
<< "\n";

static const int numThreads = 8;

sycl::buffer buffer(numThreads);
queue
.submit([&](sycl::handler &cgh) {
auto d_buffer =
buffer.get_access(cgh, sycl::write_only, sycl::no_init);
sycl::range<1> range{numThreads};
cgh.parallel_for(range, [=](sycl::id<1> threadId) {
d_buffer[threadId] = threadId.get(0);
});
})
.wait_and_throw();
const auto h_Buffer = buffer.get_access();
for (int i = 0; i < numThreads; i++) {
if (h_Buffer[i] != i) {
std::cout << "Kernel produced invalid values" << std::endl;
}
}
std::cout << "DONE!" << std::endl;
continue;
queue
.submit([&](sycl::handler &cgh) {
auto d_buffer =
buffer.get_access(cgh, sycl::write_only, sycl::no_init);
sycl::range<1> range{numThreads};
cgh.parallel_for(range, [=](sycl::id<1> threadId) {
d_buffer[threadId] = threadId.get(0) + 1.0;
});
})
.wait_and_throw();

}
}

int main() {
foo();
foo();
return 0;
}
```


Segfault location Vulkan

I have additionally enabled stack trace printing on allocation_map dtor:

```
[Thread 0x7fffb11fe6c0 (LWP 2428014) exited]
[Thread 0x7fffc8ff96c0 (LWP 2428015) exited]
### This just prints a backtrace at a breakpoint
Thread 1 "a.out" hit Breakpoint 1, 0x00007ffff7f56ed0 in hipsycl::common::allocation_map::~allocation_map() () from /tmp/AdaptiveCpp/build/clang-21-vk/install/bin/../lib/libacpp-rt.so
#0 0x00007ffff7f56ed0 in hipsycl::common::allocation_map::~allocation_map() () from /tmp/AdaptiveCpp/build/clang-21-vk/install/bin/../lib/libacpp-rt.so
#1 0x00007ffff7847a76 in __run_exit_handlers (status=0, listp=, run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at ./stdlib/exit.c:108
#2 0x00007ffff7847bbe in __GI_exit (status=) at ./stdlib/exit.c:138
#3 0x00007ffff782a1d1 in __libc_start_call_main (main=main@entry=0x55555555d730 , argc=argc@entry=1, argv=argv@entry=0x7fffffffc348) at ../sysdeps/nptl/libc_start_call_main.h:74
#4 0x00007ffff782a28b in __libc_start_main_impl (main=0x55555555d730 , argc=1, argv=0x7fffffffc348, init=, fini=, rtld_fini=, stack_end=0x7fffffffc338) at ../csu/libc-start.c:360
#5 0x000055555555ac55 in _start ()
corrupted size vs. prev_size while consolidating
### This is the actual crash when allocations are later unregistered
Thread 11 "a.out" received signal SIGABRT, Aborted.
[Switching to Thread 0x7fffc9ffb6c0 (LWP 2428009)]
__pthread_kill_implementation (no_tid=0, signo=6, threadid=) at ./nptl/pthread_kill.c:44
warning: 44 ./nptl/pthread_kill.c: No such file or directory
(gdb) bt
#0 __pthread_kill_implementation (no_tid=0, signo=6, threadid=) at ./nptl/pthread_kill.c:44
#1 __pthread_kill_internal (signo=6, threadid=) at ./nptl/pthread_kill.c:78
#2 __GI___pthread_kill (threadid=, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3 0x00007ffff784527e in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4 0x00007ffff78288ff in __GI_abort () at ./stdlib/abort.c:79
#5 0x00007ffff78297b6 in __libc_message_impl (fmt=fmt@entry=0x7ffff79ce8d7 "%s\n") at ../sysdeps/posix/libc_fatal.c:134
#6 0x00007ffff78a90d5 in malloc_printerr (str=str@entry=0x7ffff79d1b30 "corrupted size vs. prev_size while consolidating") at ./malloc/malloc.c:5775
#7 0x00007ffff78ab234 in _int_free_merge_chunk (av=0x7ffff7a03ac0 , p=0x555556453eb0, size=8544) at ./malloc/malloc.c:4695
#8 0x00007ffff78ab51a in _int_free (av=0x7ffff7a03ac0 , p=, have_lock=) at ./malloc/malloc.c:4646
#9 0x00007ffff78adeae in __GI___libc_free (mem=0x555556455f90) at ./malloc/malloc.c:3398
#10 0x00007ffff7f595ef in bool hipsycl::common::allocation_map::erase<3>(hipsycl::common::allocation_map::intermediate_node<3>&, unsigned long) ()
from /tmp/AdaptiveCpp/build/clang-21-vk/install/bin/../lib/libacpp-rt.so
#11 0x00007ffff7f59490 in bool hipsycl::common::allocation_map::erase<6>(hipsycl::common::allocation_map::intermediate_node<6>&, unsigned long) ()
from /tmp/AdaptiveCpp/build/clang-21-vk/install/bin/../lib/libacpp-rt.so
#12 0x00007ffff7f59392 in bool hipsycl::common::allocation_map::erase<9>(hipsycl::common::allocation_map::intermediate_node<9>&, unsigned long) ()
from /tmp/AdaptiveCpp/build/clang-21-vk/install/bin/../lib/libacpp-rt.so
#13 0x00007ffff7f59297 in bool hipsycl::common::allocation_map::erase<12>(hipsycl::common::allocation_map::intermediate_node<12>&, unsigned long) ()
from /tmp/AdaptiveCpp/build/clang-21-vk/install/bin/../lib/libacpp-rt.so
#14 0x00007ffff7f59190 in bool hipsycl::common::allocation_map::erase<15>(hipsycl::common::allocation_map::intermediate_node<15>&, unsigned long) ()
from /tmp/AdaptiveCpp/build/clang-21-vk/install/bin/../lib/libacpp-rt.so
#15 0x00007ffff7f56c8e in hipsycl::rt::allocation_tracker::unregister_allocation(void const*) () from /tmp/AdaptiveCpp/build/clang-21-vk/install/bin/../lib/libacpp-rt.so
#16 0x000055555556a38c in hipsycl::rt::data_region::~data_region()::{lambda(auto:1&)#1}::operator() >(hipsycl::rt::data_allocation&) const (this=this@entry=0x7fffc9ff5e8f, alloc=...) at install/bin/../include/AdaptiveCpp/hipSYCL/runtime/data.hpp:420
#17 0x000055555556a17b in hipsycl::rt::allocation_list::for_each_allocation_while::~data_region()::{lambda(auto:1&)#1}>(hipsycl::rt::data_region::~data_region()::{lambda(auto:1&)#1}&&) (this=0x555556430218, h=...) at install/bin/../include/AdaptiveCpp/hipSYCL/runtime/data.hpp:302
#18 hipsycl::rt::data_region::~data_region (this=0x555556430210) at install/bin/../include/AdaptiveCpp/hipSYCL/runtime/data.hpp:407
#19 0x00007ffff7f9a0d9 in hipsycl::rt::buffer_memory_requirement::~buffer_memory_requirement() () from /tmp/AdaptiveCpp/build/clang-21-vk/install/bin/../lib/libacpp-rt.so
#20 0x00007ffff7f9a1f9 in hipsycl::rt::buffer_memory_requirement::~buffer_memory_requirement() () from /tmp/AdaptiveCpp/build/clang-21-vk/install/bin/../lib/libacpp-rt.so
#21 0x00007ffff7f81dbb in hipsycl::rt::dag_node::~dag_node() () from /tmp/AdaptiveCpp/build/clang-21-vk/install/bin/../lib/libacpp-rt.so
#22 0x00007ffff7f5acaa in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release_last_use_cold() () from /tmp/AdaptiveCpp/build/clang-21-vk/install/bin/../lib/libacpp-rt.so
#23 0x00007ffff7f90d52 in std::_Function_handler::_M_invoke(std::_Any_data const&) () from /tmp/AdaptiveCpp/build/clang-21-vk/install/bin/../lib/libacpp-rt.so
#24 0x00007ffff7f97bdf in hipsycl::rt::worker_thread::work() () from /tmp/AdaptiveCpp/build/clang-21-vk/install/bin/../lib/libacpp-rt.so
#25 0x00007ffff7cecdb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#26 0x00007ffff789cb84 in start_thread (arg=) at ./nptl/pthread_create.c:447
#27 0x00007ffff7929d6c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

```


OpenCL segfault happens in different places, but Valgrind reliably complains a lot about `allocation_map`:

```
$ ACPP_PERSISTENT_RUNTIME=1 ACPP_VISIBILITY_MASK=opencl:A770 valgrind ./a.out
...
==2428690== Invalid read of size 8
==2428690== at 0x487E176: bool hipsycl::common::allocation_map::erase<15>(hipsycl::common::allocation_map::intermediate_node<15>&, unsigned long) (in /tmp/AdaptiveCpp/build/clang-21-vk/install/
lib/libacpp-rt.so)
==2428690== by 0x487BC8D: hipsycl::rt::allocation_tracker::unregister_allocation(void const*) (in /tmp/AdaptiveCpp/build/clang-21-vk/install/lib/libacpp-rt.so)
==2428690== by 0x11E38B: auto hipsycl::rt::data_region::~data_region()::{lambda(auto:1&)#1}::operator() >(hipsycl::rt::data_allocation&) const (data.hpp:420)
==2428690== by 0x11E17A: for_each_allocation_while<(lambda at /tmp/AdaptiveCpp/build/clang-21-vk/install/bin/../include/AdaptiveCpp/hipSYCL/runtime/data.hpp:407:44)> (data.hpp:302)
==2428690== by 0x11E17A: hipsycl::rt::data_region::~data_region() (data.hpp:407)
==2428690== by 0x48BF0D8: hipsycl::rt::buffer_memory_requirement::~buffer_memory_requirement() (in /tmp/AdaptiveCpp/build/clang-21-vk/install/lib/libacpp-rt.so)
==2428690== by 0x48BF1F8: hipsycl::rt::buffer_memory_requirement::~buffer_memory_requirement() (in /tmp/AdaptiveCpp/build/clang-21-vk/install/lib/libacpp-rt.so)
==2428690== by 0x48A6DBA: hipsycl::rt::dag_node::~dag_node() (in /tmp/AdaptiveCpp/build/clang-21-vk/install/lib/libacpp-rt.so)
==2428690== by 0x487FCA9: std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release_last_use_cold() (in /tmp/AdaptiveCpp/build/clang-21-vk/install/lib/libacpp-rt.so)
...
```

**Expected behavior**

Things not crashing

**Describe your setup**
* AdaptiveCpp 91cfc8919efdc6213e38c1bac6976b42b9be44b6, built with Clang 21.1.8

```console
$ ./install/bin/acpp-info -l
=================Backend information===================
Loaded backend 0: OpenMP
Found device: AdaptiveCpp OpenMP host device
Loaded backend 1: CUDA
Found device: NVIDIA GeForce RTX 3060
Loaded backend 2: OpenCL
Found device: Intel(R) Arc(TM) A770 Graphics
Found device: Intel(R) UHD Graphics 770
Loaded backend 3: Vulkan
Found device: Intel(R) UHD Graphics 770 (ADL-S GT1)
Found device: Intel(R) Arc(tm) A770 Graphics (DG2)
Found device: AMD Radeon RX 6400 (RADV NAVI24)
Found device: NVIDIA GeForce RTX 3060
Found device: llvmpipe (LLVM 20.1.2, 256 bits)
Loaded backend 4: HIP
Found device: AMD Radeon RX 6400
```

**Optional additional diagnostic information**

Looks like when the application is being shut down, the `static amap_t amap` gets destroyed before static `persistent_runtime_object` and thus when runtime tries to unregister its tracked allocations, things go south.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.