GoogleCloudPlatform / GoogleCloudPlatform/esp-v2

endpoints-runtime envoy startup frequently nonresponsive or segfaults on GKE

Open
#962 12 comments 14 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
307
Forks
185
Avg merge
14h 45m
Merged PRs (30d)
6

Description

Running [endpoints-runtime:2.52](https://console.cloud.google.com/artifacts/docker/endpoints-release/us/gcr.io/endpoints-runtime/sha256:d2082668e5f7e33492041458867fa46bf5f7c0bdc6ccd8e58254486470ac5bd6?inv=1&invt=Ab1p4Q) in sidecar mode on Autopilot GKE fails to reach a healthy state with increasing frequency over the past year or so.

Once the sidecar reaches a healthy state, it tends to stay healthy, even for weeks at a time. However, the transient startup issues are frequent enough now that we don't want to risk changing a deployment because the ESPv2 sidecar cannot be trusted to reach a healthy state.

More and more often, our deployment logs segmentation faults from the Envoy process, as seen in the image.

Image

The instances of segmentation faults do not have consistent backtraces across instances.

I know very little about Envoy, but FWIW, https://github.com/envoyproxy/envoy/issues/27775 expresses similar symptoms with a suggested root cause.

Is this a known issue? Any help would be appreciated.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.