envoyproxy / envoyproxy/envoy

Envoy Windows crashes on startup due to access violation (0xc0000005)

Open
#34,545 7 comments 0 reactions 0 assignees View on GitHub
area/windows bug no stalebot
Dominant language
C++
Stars
28.9k
Forks
5.6k
Avg merge
1d 22h
Merged PRs (30d)
430

Description

*Note: this issue was previously reported to envoy-security@googlegroups.com and approved for posting publicly*

We run Envoy on Windows desktops, and a user is encountering unexpected crashes as envoy.exe starts up. It is difficult for us to reproduce locally, but it is easily repeatable on their machine(s).

We are currently on 1.27.0 and using the precompiled binary made available via the docker image, as recommended [here](https://github.com/envoyproxy/envoy/issues/5668#issuecomment-456129910). The issue appears to happen on 1.28 (the latest Windows release) as well.

What we have observed is Envoy exits immediately with a 3221225477 (hex: 0xc0000005) [access violation](https://stackoverflow.com/q/5303524/113632) exit status, with no output to stdout/err.

We next tried running the envoy binary by hand, and observed it crashes even with trivial arguments like `--version` or incorrect arguments that should have caused an argument parsing error. We found a `.wer` error log file, the salient details of which are:

```
Sig[0].Name=Application name
Sig[0].Value=envoy.exe
Sig[1].Name=Application version
Sig[1].Value=0.0.0.0
Sig[2].Name=Application time stamp
Sig[2].Value=64c15a94
Sig[3].Name=Failure Module Name
Sig[3].Value=ntdll.dll
Sig[4].Name=Failure Module Name version
Sig[4].Value=10.0.19041.3996
Sig[5].Name=Failure Module Name time stamp
Sig[5].Value=39215800
Sig[6].Name=Exception code
Sig[6].Value=c0000005
Sig[7].Name=Exception offset
Sig[7].Value=00000000000634f6
```

The user has CrowdStrike installed on their machine, and disabling this appears to allow Envoy to run as expected, so we _think_ this is a compatibility issue with CrowdStrike. Some Googling found several relevant-seeming discussions about CrowdStrike causing unexpected crashes and incompatibilities, including:

* https://resource.dopus.com/t/crashes-possibly-due-to-crowdstrike-falcon-sensor/41852
* https://knowledge.broadcom.com/external/article/176409/applications-do-not-launch-when-applicat.html
* https://forum.eset.com/topic/40242-deep-behavioral-inspection-causes-cmd-crash/
* https://kcm.trellix.com/corporate/index?page=content&id=KB91225

I realize that Windows support is mothballed at the moment, so my primary motivation for filing this bug is to aid discovery should anyone else encounter a similar crash. Should Windows support be restored it would be helpful if Envoy was able to detect this situation and fail more gracefully than the current access violation crashes.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.