AdaptiveCpp / AdaptiveCpp/AdaptiveCpp
allocation_map lifetime issues with ACPP_PERSISTENT_RUNTIME
- Dominant language
- C++
- Stars
- 1.9k
- Forks
- 228
- Avg merge
- 4d 1h
- Merged PRs (30d)
- 25
Description
**Bug summary**
When `ACPP_PERSISTENT_RUNTIME=1` is set, the application sporadically segfaults at shutdown when a backend with allocation tracking is used (verified with OpenCL and Vulkan).
**To Reproduce**
The example below most reliably triggers the failure. I've seen it even with a single `foo` invocation, but is takes longer to manifest.
```console
$ acpp sycl-app-reinit.cpp --acpp-targets=generic -O3 -g
$ i=0; while true; do i=$(($i+1)); ACPP_PERSISTENT_RUNTIME=1 ACPP_VISIBILITY_MASK=opencl:A770 ./a.out &>/dev/null || break; done; echo "Crashed after $i repeats"
Aborted (core dumped)
Crashed after 17 repeats
$ i=0; while true; do i=$(($i+1)); ACPP_PERSISTENT_RUNTIME=1 ACPP_VISIBILITY_MASK=vk:A770 ./a.out &>/dev/null || break; done; echo "Crashed after $i repeats"
Segmentation fault (core dumped)
Crashed after 6 repeats
$ i=0; while true; do i=$(($i+1)); ACPP_PERSISTENT_RUNTIME=1 ACPP_VISIBILITY_MASK=vk:NVIDIA ./a.out &>/dev/null || break; done; echo "Crashed after $i repeats"
Segmentation fault (core dumped)
Crashed after 4 repeats
```
sycl-app-reinit.cpp
```cpp
#include
#include
class Kernel;
void foo() {
// Creating buffer of 4 elements to be used inside the kernel code
sycl::buffer Buffer(4);
for (const auto &Device : sycl::device::get_devices()) {
// Creating SYCL queue
//sycl::queue queue(Device, {sycl::property::queue::in_order()});
sycl::queue queue{Device, {sycl::property::queue::AdaptiveCpp_priority{1}, sycl::property::queue::in_order()}};
std::cout << "Running on "
<< queue.get_device().get_info()
<< "\n";
static const int numThreads = 8;
sycl::buffer buffer(numThreads);
queue
.submit([&](sycl::handler &cgh) {
auto d_buffer =
buffer.get_access(cgh, sycl::write_only, sycl::no_init);
sycl::range<1> range{numThreads};
cgh.parallel_for(range, [=](sycl::id<1> threadId) {
d_buffer[threadId] = threadId.get(0);
});
})
.wait_and_throw();
const auto h_Buffer = buffer.get_access();
for (int i = 0; i < numThreads; i++) {
if (h_Buffer[i] != i) {
std::cout << "Kernel produced invalid values" << std::endl;
}
}
std::cout << "DONE!" << std::endl;
continue;
queue
.submit([&](sycl::handler &cgh) {
auto d_buffer =
buffer.get_access(cgh, sycl::write_only, sycl::no_init);
sycl::range<1> range{numThreads};
cgh.parallel_for(range, [=](sycl::id<1> threadId) {
d_buffer[threadId] = threadId.get(0) + 1.0;
});
})
.wait_and_throw();
}
}
int main() {
foo();
foo();
return 0;
}
```
Segfault location Vulkan
I have additionally enabled stack trace printing on allocation_map dtor:
```
[Thread 0x7fffb11fe6c0 (LWP 2428014) exited]
[Thread 0x7fffc8ff96c0 (LWP 2428015) exited]
### This just prints a backtrace at a breakpoint
Thread 1 "a.out" hit Breakpoint 1, 0x00007ffff7f56ed0 in hipsycl::common::allocation_map::~allocation_map() () from /tmp/AdaptiveCpp/build/clang-21-vk/install/bin/../lib/libacpp-rt.so
#0 0x00007ffff7f56ed0 in hipsycl::common::allocation_map::~allocation_map() () from /tmp/AdaptiveCpp/build/clang-21-vk/install/bin/../lib/libacpp-rt.so
#1 0x00007ffff7847a76 in __run_exit_handlers (status=0, listp=, run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at ./stdlib/exit.c:108
#2 0x00007ffff7847bbe in __GI_exit (status=) at ./stdlib/exit.c:138
#3 0x00007ffff782a1d1 in __libc_start_call_main (main=main@entry=0x55555555d730 , argc=argc@entry=1, argv=argv@entry=0x7fffffffc348) at ../sysdeps/nptl/libc_start_call_main.h:74
#4 0x00007ffff782a28b in __libc_start_main_impl (main=0x55555555d730 , argc=1, argv=0x7fffffffc348, init=, fini=, rtld_fini=, stack_end=0x7fffffffc338) at ../csu/libc-start.c:360
#5 0x000055555555ac55 in _start ()
corrupted size vs. prev_size while consolidating
### This is the actual crash when allocations are later unregistered
Thread 11 "a.out" received signal SIGABRT, Aborted.
[Switching to Thread 0x7fffc9ffb6c0 (LWP 2428009)]
__pthread_kill_implementation (no_tid=0, signo=6, threadid=) at ./nptl/pthread_kill.c:44
warning: 44 ./nptl/pthread_kill.c: No such file or directory
(gdb) bt
#0 __pthread_kill_implementation (no_tid=0, signo=6, threadid=) at ./nptl/pthread_kill.c:44
#1 __pthread_kill_internal (signo=6, threadid=) at ./nptl/pthread_kill.c:78
#2 __GI___pthread_kill (threadid=, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3 0x00007ffff784527e in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4 0x00007ffff78288ff in __GI_abort () at ./stdlib/abort.c:79
#5 0x00007ffff78297b6 in __libc_message_impl (fmt=fmt@entry=0x7ffff79ce8d7 "%s\n") at ../sysdeps/posix/libc_fatal.c:134
#6 0x00007ffff78a90d5 in malloc_printerr (str=str@entry=0x7ffff79d1b30 "corrupted size vs. prev_size while consolidating") at ./malloc/malloc.c:5775
#7 0x00007ffff78ab234 in _int_free_merge_chunk (av=0x7ffff7a03ac0 , p=0x555556453eb0, size=8544) at ./malloc/malloc.c:4695
#8 0x00007ffff78ab51a in _int_free (av=0x7ffff7a03ac0 , p=, have_lock=) at ./malloc/malloc.c:4646
#9 0x00007ffff78adeae in __GI___libc_free (mem=0x555556455f90) at ./malloc/malloc.c:3398
#10 0x00007ffff7f595ef in bool hipsycl::common::allocation_map::erase<3>(hipsycl::common::allocation_map::intermediate_node<3>&, unsigned long) ()
from /tmp/AdaptiveCpp/build/clang-21-vk/install/bin/../lib/libacpp-rt.so
#11 0x00007ffff7f59490 in bool hipsycl::common::allocation_map::erase<6>(hipsycl::common::allocation_map::intermediate_node<6>&, unsigned long) ()
from /tmp/AdaptiveCpp/build/clang-21-vk/install/bin/../lib/libacpp-rt.so
#12 0x00007ffff7f59392 in bool hipsycl::common::allocation_map::erase<9>(hipsycl::common::allocation_map::intermediate_node<9>&, unsigned long) ()
from /tmp/AdaptiveCpp/build/clang-21-vk/install/bin/../lib/libacpp-rt.so
#13 0x00007ffff7f59297 in bool hipsycl::common::allocation_map::erase<12>(hipsycl::common::allocation_map::intermediate_node<12>&, unsigned long) ()
from /tmp/AdaptiveCpp/build/clang-21-vk/install/bin/../lib/libacpp-rt.so
#14 0x00007ffff7f59190 in bool hipsycl::common::allocation_map::erase<15>(hipsycl::common::allocation_map::intermediate_node<15>&, unsigned long) ()
from /tmp/AdaptiveCpp/build/clang-21-vk/install/bin/../lib/libacpp-rt.so
#15 0x00007ffff7f56c8e in hipsycl::rt::allocation_tracker::unregister_allocation(void const*) () from /tmp/AdaptiveCpp/build/clang-21-vk/install/bin/../lib/libacpp-rt.so
#16 0x000055555556a38c in hipsycl::rt::data_region::~data_region()::{lambda(auto:1&)#1}::operator() >(hipsycl::rt::data_allocation&) const (this=this@entry=0x7fffc9ff5e8f, alloc=...) at install/bin/../include/AdaptiveCpp/hipSYCL/runtime/data.hpp:420
#17 0x000055555556a17b in hipsycl::rt::allocation_list::for_each_allocation_while::~data_region()::{lambda(auto:1&)#1}>(hipsycl::rt::data_region::~data_region()::{lambda(auto:1&)#1}&&) (this=0x555556430218, h=...) at install/bin/../include/AdaptiveCpp/hipSYCL/runtime/data.hpp:302
#18 hipsycl::rt::data_region::~data_region (this=0x555556430210) at install/bin/../include/AdaptiveCpp/hipSYCL/runtime/data.hpp:407
#19 0x00007ffff7f9a0d9 in hipsycl::rt::buffer_memory_requirement::~buffer_memory_requirement() () from /tmp/AdaptiveCpp/build/clang-21-vk/install/bin/../lib/libacpp-rt.so
#20 0x00007ffff7f9a1f9 in hipsycl::rt::buffer_memory_requirement::~buffer_memory_requirement() () from /tmp/AdaptiveCpp/build/clang-21-vk/install/bin/../lib/libacpp-rt.so
#21 0x00007ffff7f81dbb in hipsycl::rt::dag_node::~dag_node() () from /tmp/AdaptiveCpp/build/clang-21-vk/install/bin/../lib/libacpp-rt.so
#22 0x00007ffff7f5acaa in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release_last_use_cold() () from /tmp/AdaptiveCpp/build/clang-21-vk/install/bin/../lib/libacpp-rt.so
#23 0x00007ffff7f90d52 in std::_Function_handler::_M_invoke(std::_Any_data const&) () from /tmp/AdaptiveCpp/build/clang-21-vk/install/bin/../lib/libacpp-rt.so
#24 0x00007ffff7f97bdf in hipsycl::rt::worker_thread::work() () from /tmp/AdaptiveCpp/build/clang-21-vk/install/bin/../lib/libacpp-rt.so
#25 0x00007ffff7cecdb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#26 0x00007ffff789cb84 in start_thread (arg=) at ./nptl/pthread_create.c:447
#27 0x00007ffff7929d6c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
```
OpenCL segfault happens in different places, but Valgrind reliably complains a lot about `allocation_map`:
```
$ ACPP_PERSISTENT_RUNTIME=1 ACPP_VISIBILITY_MASK=opencl:A770 valgrind ./a.out
...
==2428690== Invalid read of size 8
==2428690== at 0x487E176: bool hipsycl::common::allocation_map::erase<15>(hipsycl::common::allocation_map::intermediate_node<15>&, unsigned long) (in /tmp/AdaptiveCpp/build/clang-21-vk/install/
lib/libacpp-rt.so)
==2428690== by 0x487BC8D: hipsycl::rt::allocation_tracker::unregister_allocation(void const*) (in /tmp/AdaptiveCpp/build/clang-21-vk/install/lib/libacpp-rt.so)
==2428690== by 0x11E38B: auto hipsycl::rt::data_region::~data_region()::{lambda(auto:1&)#1}::operator() >(hipsycl::rt::data_allocation&) const (data.hpp:420)
==2428690== by 0x11E17A: for_each_allocation_while<(lambda at /tmp/AdaptiveCpp/build/clang-21-vk/install/bin/../include/AdaptiveCpp/hipSYCL/runtime/data.hpp:407:44)> (data.hpp:302)
==2428690== by 0x11E17A: hipsycl::rt::data_region::~data_region() (data.hpp:407)
==2428690== by 0x48BF0D8: hipsycl::rt::buffer_memory_requirement::~buffer_memory_requirement() (in /tmp/AdaptiveCpp/build/clang-21-vk/install/lib/libacpp-rt.so)
==2428690== by 0x48BF1F8: hipsycl::rt::buffer_memory_requirement::~buffer_memory_requirement() (in /tmp/AdaptiveCpp/build/clang-21-vk/install/lib/libacpp-rt.so)
==2428690== by 0x48A6DBA: hipsycl::rt::dag_node::~dag_node() (in /tmp/AdaptiveCpp/build/clang-21-vk/install/lib/libacpp-rt.so)
==2428690== by 0x487FCA9: std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release_last_use_cold() (in /tmp/AdaptiveCpp/build/clang-21-vk/install/lib/libacpp-rt.so)
...
```
**Expected behavior**
Things not crashing
**Describe your setup**
* AdaptiveCpp 91cfc8919efdc6213e38c1bac6976b42b9be44b6, built with Clang 21.1.8
```console
$ ./install/bin/acpp-info -l
=================Backend information===================
Loaded backend 0: OpenMP
Found device: AdaptiveCpp OpenMP host device
Loaded backend 1: CUDA
Found device: NVIDIA GeForce RTX 3060
Loaded backend 2: OpenCL
Found device: Intel(R) Arc(TM) A770 Graphics
Found device: Intel(R) UHD Graphics 770
Loaded backend 3: Vulkan
Found device: Intel(R) UHD Graphics 770 (ADL-S GT1)
Found device: Intel(R) Arc(tm) A770 Graphics (DG2)
Found device: AMD Radeon RX 6400 (RADV NAVI24)
Found device: NVIDIA GeForce RTX 3060
Found device: llvmpipe (LLVM 20.1.2, 256 bits)
Loaded backend 4: HIP
Found device: AMD Radeon RX 6400
```
**Optional additional diagnostic information**
Looks like when the application is being shut down, the `static amap_t amap` gets destroyed before static `persistent_runtime_object` and thus when runtime tries to unregister its tracked allocations, things go south.
Contributor guide
Assessment
This issue has not been assessed yet.