envoyproxy / envoyproxy/envoy

Crash in Envoy::Http::AsyncStreamImpl::closeRemote on shutdown

Open
#7,983 3 comments 0 reactions 0 assignees View on GitHub
bug help wanted
Dominant language
C++
Stars
28.9k
Forks
5.6k
Avg merge
1d 22h
Merged PRs (30d)
430

Description

Seeing the following crash running off 38a4d86407668612ccbc4cba65d11be6cbd81b32:

```
2019-08-20_22:24:17.36421 [2019-08-20 22:24:17.364][374905][warning][main] [external/envoy/source/server/server.cc:469] caught SIGTERM
2019-08-20_22:24:17.36423 [2019-08-20 22:24:17.364][374905][info][main] [external/envoy/source/server/server.cc:574] shutting down server instance
2019-08-20_22:24:17.36424 [2019-08-20 22:24:17.364][374905][info][main] [external/envoy/source/server/server.cc:528] main dispatch loop exited
2019-08-20_22:24:17.36959 [2019-08-20 22:24:17.369][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:81] Caught Segmentation f
ault, suspect faulting address 0x0
2019-08-20_22:24:17.36962 [2019-08-20 22:24:17.369][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:69] Backtrace (use tools/
stack_decode.py to get line numbers):
2019-08-20_22:24:17.37594 [2019-08-20 22:24:17.375][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #0: __restore_rt [0x7
f1f290895d0]
2019-08-20_22:24:17.39615 [2019-08-20 22:24:17.396][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #1: Envoy::Http::AsyncStreamImpl::closeRemote() [0x8f0b46]
2019-08-20_22:24:17.40303 [2019-08-20 22:24:17.402][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #2: Envoy::Http::AsyncStreamImpl::encodeData() [0x8f5148]
2019-08-20_22:24:17.41097 [2019-08-20 22:24:17.410][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #3: Envoy::Http::Utility::sendLocalReply() [0xa05ff8]
2019-08-20_22:24:17.41454 [2019-08-20 22:24:17.414][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #4: Envoy::Http::AsyncStreamImpl::sendLocalReply() [0x8f0e69]
2019-08-20_22:24:17.41906 [2019-08-20 22:24:17.419][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #5: Envoy::Router::Filter::onUpstreamAbort() [0x8f71c3]
2019-08-20_22:24:17.42254 [2019-08-20 22:24:17.422][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #6: Envoy::Router::Filter::onUpstreamReset() [0x902bee]
2019-08-20_22:24:17.42603 [2019-08-20 22:24:17.425][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #7: Envoy::Router::Filter::UpstreamRequest::onResetStream() [0x902fb2]
2019-08-20_22:24:17.42950 [2019-08-20 22:24:17.429][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #8: Envoy::Http::Http1::ClientConnectionImpl::onResetStream() [0x95f30d]
2019-08-20_22:24:17.43457 [2019-08-20 22:24:17.434][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #9: Envoy::Http::CodecClient::onEvent() [0x8e27a4]
2019-08-20_22:24:17.44230 [2019-08-20 22:24:17.442][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #10: Envoy::Network::ConnectionImpl::raiseEvent() [0x753b25]
2019-08-20_22:24:17.44620 [2019-08-20 22:24:17.446][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #11: Envoy::Network::ConnectionImpl::closeSocket() [0x753045]
2019-08-20_22:24:17.45422 [2019-08-20 22:24:17.454][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #12: Envoy::Network::ConnectionImpl::close() [0x75489d]
2019-08-20_22:24:17.45843 [2019-08-20 22:24:17.458][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #13: Envoy::Http::Http1::ConnPoolImpl::~ConnPoolImpl() [0x88abf5]
2019-08-20_22:24:17.46249 [2019-08-20 22:24:17.462][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #14: Envoy::Http::Http1::ProdConnPoolImpl::~ProdConnPoolImpl() [0x88ad00]
2019-08-20_22:24:17.46651 [2019-08-20 22:24:17.466][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #15: Envoy::Upstream::ConnPoolMap<>::clearActivePools() [0x78500e]
2019-08-20_22:24:17.46989 [2019-08-20 22:24:17.469][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #16: Envoy::Upstream::ConnPoolMap<>::~ConnPoolMap() [0x78c2e5]
2019-08-20_22:24:17.47316 [2019-08-20 22:24:17.473][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #17: std::_Sp_counted_ptr_inplace<>::_M_dispose() [0x78c428]
2019-08-20_22:24:17.47640 [2019-08-20 22:24:17.476][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #18: std::_Hashtable<>::clear() [0x784e4a]
2019-08-20_22:24:17.47980 [2019-08-20 22:24:17.479][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #19: Envoy::Upstream::ClusterManagerImpl::ThreadLocalClusterManagerImpl::~ThreadLocalClusterManagerImpl() [0x789c2b]
2019-08-20_22:24:17.48601 [2019-08-20 22:24:17.485][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #20: Envoy::ThreadLoc
al::InstanceImpl::shutdownThread() [0x70873a]
2019-08-20_22:24:17.48990 [2019-08-20 22:24:17.489][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #21: Envoy::Server::W
orkerImpl::threadRoutine() [0x746d49]
2019-08-20_22:24:17.49403 [2019-08-20 22:24:17.493][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #22: Envoy::Thread::T
hreadImplPosix::ThreadImplPosix()::{lambda()#1}::_FUN() [0xc879c5]
2019-08-20_22:24:17.49410 [2019-08-20 22:24:17.494][376231][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #23: start_thread [0x
7f1f29081dd5]
2019-08-20_22:24:17.49412 AsyncClient 0x4981808, stream_id_: 1003858367269869698
2019-08-20_22:24:17.49413 &stream_info_:
2019-08-20_22:24:17.49413 StreamInfoImpl 0x4981960, protocol_: 1, response_code_: null, response_code_details_: upstream_reset_before_response_started{connection termination}, health_check_request_: 0, rout
e_name_:
```

Not 100% sure that this was introduced with our recent update, as we've been seeing crashes with similar traces on older versions.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.