google / google/googletest

[Bug]: Example __asan_on_error hangs when triggered

Open
#4,830 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
39.5k
Forks
10.9k
Avg merge
6d 13h
Merged PRs (30d)
1

Description

### Describe the issue

Hi, I'm trying to run a simple GTest example with Address Sanitizer (asan) enabled on Windows 11 using Visual Studio 17.14.13 and GTest 1.17.0. I'm installing GTest through vcpkg, and using a custom vcpkg triplet to make sure it's built with asan enabled as well.

I'm trying to see how the `__asan_on_error` function sample from https://google.github.io/googletest/advanced.html#sanitizer-integration works in practice. However, when I reach this function the test program hangs entirely.

### Steps to reproduce the problem

My code is as follows:

vcpkg.json:
```
{
"$schema": "https://raw.githubusercontent.com/microsoft/vcpkg/master/scripts/vcpkg.schema.json",
"name": "gtest-asan",
"version-string": "1.0.0.0",
"builtin-baseline": "dd3097e305afa53f7b4312371f62058d2e665320",
"dependencies": [
"gtest"
]
}
```

CMakeLists.txt:
```
cmake_minimum_required(VERSION 3.29.0 FATAL_ERROR)

set(CMAKE_TOOLCHAIN_FILE
"C:/Users/martin/vcpkg/scripts/buildsystems/vcpkg.cmake" CACHE PATH ""
)

set(VCPKG_BOOTSTRAP_OPTIONS "-disableMetrics")
set(VCPKG_OVERLAY_TRIPLETS "${CMAKE_CURRENT_SOURCE_DIR}")
set(VCPKG_TARGET_TRIPLET "x64-windows-asan")

project(gtest_asan CXX)

find_package(GTest CONFIG REQUIRED)

add_executable(gtest_asan
gtest_asan.cpp
)

target_compile_options(gtest_asan PRIVATE
/fsanitize=address
/Zi
)

target_link_options(gtest_asan PRIVATE
/INCREMENTAL:NO
)

target_link_libraries(gtest_asan PRIVATE
GTest::gtest_main
)
```

x64-windows-asan.cmake:
```
set(VCPKG_TARGET_ARCHITECTURE x64)
set(VCPKG_CRT_LINKAGE dynamic)
set(VCPKG_LIBRARY_LINKAGE dynamic)

set(VCPKG_C_FLAGS "/fsanitize=address")
set(VCPKG_CXX_FLAGS "/fsanitize=address")
set(VCPKG_LINKER_FLAGS "/INCREMENTAL:NO")
```

gtest_asan.cpp:
```
#include

extern "C" {

void __asan_on_error()
{
FAIL() << "Encountered an address sanitizer error";
}

}

char get_char()
{
char* p = new char[42];

delete p;

return p[5];
}

TEST(MyTest, MyTest)
{
EXPECT_EQ('a', get_char());
}

int main(int argc, char** argv)
{
::testing::InitGoogleTest(&argc, argv);

return RUN_ALL_TESTS();
}
```

To build:
```
cmake -B build -GNinja -DCMAKE_BUILD_TYPE=Release
cmake --build build
```

When running this test without `__asan_on_error` I can see the correct asan output and the test program finishes immediately after. However, when using this `__asan_on_error` I see that the test program hangs forever. Leaving the function empty seems to "fix" this, so my thought is that `FAIL()` isn't working as it should in this context.

Am I doing something wrong here? Thanks.

### What version of GoogleTest are you using?

1.17.0

### What operating system and version are you using?

Windows 11

### What compiler and version are you using?

Visual Studio 17.14.13
Microsoft (R) C/C++ Optimizing Compiler Version 19.44.35215 for x64

### What build system are you using?

cmake version 3.31.5

Contributor guide

Open the contributing guide

Research direction

Start with the sanitizer integration section in advanced.html and the reproducer in gtest_asan.cpp, especially __asan_on_error, FAIL(), and RUN_ALL_TESTS(). Build and run the CMake/Ninja project with the supplied AddressSanitizer options, then compare the behavior with an empty callback. Done means identifying whether FAIL() is supported in this callback context and documenting or correcting the behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
cmake, cpp
Domain
testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.