grpc / grpc/grpc

Crash during process shutdown on Windows

Open
#43,440 0 comments 0 reactions 1 assignee Claimed by @pawbhard View on GitHub
kind/bug lang/c++ priority/P2 untriaged
Dominant language
C++
Stars
45.3k
Forks
11.4k
Avg merge
2d 12h
Merged PRs (30d)
47

Description

### What version of gRPC and what language are you using?

Language: C++

gRPC: 1.81.1 \
vcpkg baseline: `9e593bb18ea69cc5095e012465dcd675a822ed0d`

The same behavior was also reproduced with gRPC 1.76.0.

### What operating system (Linux, Windows,...) and version?

Windows 10, build 19045.6456

### What runtime / compiler are you using?

Visual Studio: 2022 17.14.40 \
MSVC compiler: 19.44.35228 \
MSVC toolset: 14.44.35207 \
Windows SDK: 10.0.26100.0

vcpkg triplet: `x64-windows-static`

Configuration: Debug \
C++ standard: C++20 \
MSVC runtime: `/MTd`

Relevant compiler options:

```text
/Od /Ob0 /RTC1 /Zi
```

Relevant linker options:

```text
/machine:x64 /debug /INCREMENTAL /subsystem:console
```

### What did you do?

I created a minimal standalone C++ application that:

1. Starts a gRPC server on `127.0.0.1:0`.
2. Creates a client channel to the selected port.
3. Performs one unary RPC.
4. Destroys the client channel.
5. Calls:

```cpp
server->Shutdown();
server->Wait();
```

6. Returns from `main()`.

A minimal reproducer is attached.

Contents of the attached issue.zip file:

```text
main.cpp
greet.proto
greet.pb.h
greet.pb.cc
greet.grpc.pb.h
greet.grpc.pb.cc
```

The generated protobuf/gRPC sources are included so the reproducer can be built without regenerating them.

The crash is timing-dependent and is easiest to observe under the Visual Studio debugger.

### What did you expect to see?

The process should terminate normally after all application-owned gRPC objects have been destroyed and after:

```cpp
server->Shutdown();
server->Wait();
```

No gRPC worker thread should crash during CRT/STL shutdown.

### What did you see instead?

Sometimes, during process shutdown, a gRPC worker thread crashes with:

```text
0xC0000005: Access violation
```

The worker thread is typically inside MSVC Debug STL synchronization while cleaning up gRPC objects, for example:

```text
EnterCriticalSection
_Mtxlock
std::_Lockit::_Lockit
std::_Container_base12::_Orphan_all
...
grpc_event_engine::experimental::WorkStealingThreadPool::ThreadState::ThreadBody
```

The specific gRPC object being destroyed may vary between runs.

At the same time, the main thread is already inside MSVC CRT shutdown, for example:

```text
_RTC_Shutdown
_RTC_Terminate
_execute_onexit_table
common_exit
exit
```

At the time of the crash, gRPC cleanup is still running on a worker thread while process teardown has already started on the main thread.

### Workaround and possible cause

Explicitly keeping the default EventEngine alive for the entire application lifetime and shutting it down before returning from `main()` makes the reproducer stable:

```cpp
namespace ee = grpc_event_engine::experimental;

int main()
{
ee::SetDefaultEventEngine(ee::CreateEventEngine());

{
// Create server and channel, perform RPC,
// destroy channel, then Shutdown()/Wait() the server.
}

ee::ShutdownDefaultEventEngine();

return 0;
}
```

With this workaround I have not been able to reproduce the crash.

Calling only:

```cpp
grpc_event_engine::experimental::ShutdownDefaultEventEngine();
```

after the gRPC objects had already been destroyed did not reliably prevent the crash.

This suggests that the default EventEngine lifetime may be related to the problem, but this is only a hypothesis based on the observed stack traces and workaround behavior.

[issue.zip](https://github.com/user-attachments/files/32261407/issue.zip)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.