envoyproxy / envoyproxy/envoy

Envoy crash when validating bootstrap config file

Open
#27,775 16 comments 0 reactions 0 assignees View on GitHub
area/bootstrap bug no stalebot
Dominant language
C++
Stars
28.9k
Forks
5.6k
Avg merge
1d 20h
Merged PRs (30d)
437

Description

Description:
Envoy occasionally crashes when validating bootstrap config

Repro steps:
This has only been observed on 0.05% of Stripe hosts that run Envoy. And I can login to those hosts and manually call and roughly get segfault 1 out of 2/3 times

$ sudo /pay/jenkins-artifacts/envoy/1.24.4-stripe1/envoy-stripe --config-path $bootstrap_path --mode validate --service-cluster certhorse --service-zone us-west-2b --service-node qa-certhorse--01b66f3177b5f6cdc
Segmentation fault
Or sometimes the validate is hanging indefinitely.

Call stack:

[external/envoy/source/server/backtrace.h:104] Caught Segmentation fault, suspect faulting address 0x0
[external/envoy/source/server/backtrace.h:91] Backtrace (use tools/stack_decode.py to get line numbers):
[external/envoy/source/server/backtrace.h:92] Envoy version: 2f44165e55dd47475c44d2d03018eac3cb8a6264/1.24.4-stripe1/Clean/RELEASE/BoringSSL
[external/envoy/source/server/backtrace.h:96] #0: __restore_rt [0x7f26c19d1420]
[external/envoy/source/server/backtrace.h:96] #1: tcmalloc::tcmalloc_internal::cpu_cache_internal::CpuCache<>::Refill() [0x55cf3f28ce2a]
[external/envoy/source/server/backtrace.h:96] #2: tcmalloc::tcmalloc_internal::cpu_cache_internal::CpuCache<>::Allocate<>()::Helper::Underflow() [0x55cf3f28df77]
[external/envoy/source/server/backtrace.h:96] #3: Envoy::Api::ValidationImpl::allocateDispatcher() [0x55cf3dd5b3de]
[external/envoy/source/server/backtrace.h:96] #4: Envoy::Server::ValidationInstance::ValidationInstance() [0x55cf3dd4b52f]
[external/envoy/source/server/backtrace.h:96] #5: Envoy::Server::validateConfig() [0x55cf3dd4aa65]
[external/envoy/source/server/backtrace.h:96] #6: Envoy::MainCommonBase::run() [0x55cf3dd11370]
[external/envoy/source/server/backtrace.h:96] #7: Envoy::MainCommon::main() [0x55cf3dd11a7d]
[external/envoy/source/server/backtrace.h:96] #8: main [0x55cf3dd0da4a]
[external/envoy/source/server/backtrace.h:96] #9: __libc_start_main [0x7f26c17ef083]
uname -a:
Linux 5.15.0-1036-aws https://github.com/envoyproxy/go-control-plane/issues/40~20.04.1-Ubuntu SMP Mon Apr 24 00:21:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Envoy version
2f44165e55dd47475c44d2d03018eac3cb8a6264/1.24.4-stripe1/Clean/RELEASE/BoringSSL

2f44165e55dd47475c44d2d03018eac3cb8a6264 is internal commit of Stripe's Envoy repo, it uses OSS envoy 1.24.4

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.