envoyproxy / envoyproxy/envoy

Envoy sometime crash with TTL in xDS

Open
#20,131 4 comments 0 reactions 1 assignee Claimed by @snowp View on GitHub
area/xds bug help wanted investigate
Dominant language
C++
Stars
28.9k
Forks
5.6k
Avg merge
1d 20h
Merged PRs (30d)
428

Description

We use [envoy-control-plane](https://github.com/maksim-paskal/envoy-control-plane) to push xDS messages to envoy sidecars in our kubernetes cluster. Sometime envoy sidecar crash with this critical exception - this happens in different PODS with different configuration in different time.

```
#0 0x00007fa9c74d6817 in raise () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00005614aeffd5c6 in Envoy::SignalAction::sigHandler (sig=6, info=0x7fa9c80764b0, context=0x7fa9c8076380) at source/common/signal/signal_action.cc:53
#2
#3 0x00007fa9c7111fb7 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#4 0x00007fa9c7113921 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#5 0x00005614af150f1b in Envoy::TerminateHandler::logOnTerminate() const::$_0::operator()() const (this=)
at source/exe/terminate_handler.cc:14
#6 0x00005614af150dd9 in Envoy::TerminateHandler::logOnTerminate() const::$_0::__invoke() () at source/exe/terminate_handler.cc:11
#7 0x00005614af601433 in std::__terminate(void (*)()) ()
#8 0x00005614af6013e8 in std::terminate() ()
#9 0x00005614ad438adb in __clang_call_terminate ()
#10 0x00005614aecd2443 in Envoy::Config::TtlManager::ScopedTtlUpdate::~ScopedTtlUpdate (this=this@entry=0x7ffd10346a08) at ./source/common/config/ttl.h:35
#11 0x00005614aecd27a3 in Envoy::Config::TtlManager::TtlManager(std::__1::function, std::__1::allocator >, std::__1::allocator, std::__1::allocator > > > const&)>, Envoy::Event::Dispatcher&, Envoy::TimeSource&)::$_0::operator()() const (this=) at source/common/config/ttl.cc:30
#12 _ZNSt3__18__invokeIRZN5Envoy6Config10TtlManagerC1ENS_8functionIFvRKNS_6vectorINS_12basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEENS9_ISB_EEEEEEERNS1_5Event10DispatcherERNS1_10TimeSourceEE3$_0JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOSP_DpOSQ_ (__f=...)
at /opt/llvm/bin/../include/c++/v1/type_traits:3694
#13 std::__1::__invoke_void_return_wrapper::__call, std::__1::allocator >, std::__1::allocator, std::__1::allocator > > > const&)>, Envoy::Event::Dispatcher&, Envoy::TimeSource&)::$_0&>(Envoy::Config::TtlManager::TtlManager(std::__1::function, std::__1::allocator >, std::__1::allocator, std::__1::allocator > > > const&)>, Envoy::Event::Dispatcher&, Envoy::TimeSource&)::$_0&) (__args=...)
at /opt/llvm/bin/../include/c++/v1/__functional_base:348
#14 _ZNSt3__110__function12__alloc_funcIZN5Envoy6Config10TtlManagerC1ENS_8functionIFvRKNS_6vectorINS_12basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEENSA_ISC_EEEEEEERNS2_5Event10DispatcherERNS2_10TimeSourceEE3$_0NSA_ISO_EEFvvEEclEv (this=) at /opt/llvm/bin/../include/c++/v1/functional:1558
#15 _ZNSt3__110__function6__funcIZN5Envoy6Config10TtlManagerC1ENS_8functionIFvRKNS_6vectorINS_12basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEENSA_ISC_EEEEEEERNS2_5Event10DispatcherERNS2_10TimeSourceEE3$_0NSA_ISO_EEFvvEEclEv (this=) at /opt/llvm/bin/../include/c++/v1/functional:1732
#16 0x00005614aeff1220 in event_process_active_single_queue ()
#17 0x00005614aefeff11 in event_base_loop ()
#18 0x00005614ae85e61c in Envoy::Server::InstanceImpl::run (this=0x54e37fa92f00) at source/server/server.cc:872
#19 0x00005614ad438d64 in Envoy::MainCommonBase::run (this=0x54e37fd0b170) at source/exe/main_common.cc:173
#20 0x00005614ad4395d6 in Envoy::MainCommon::run (this=) at ./source/exe/main_common.h:100
#21 Envoy::MainCommon::main(int, char**, std::__1::function) (argc=11, argv=0x7ffd10346ee8, hook=...)
at source/exe/main_common.cc:256
#22 0x00005614ad43579c in main (argc=2, argv=0x7ffd103464e0) at source/exe/main.cc:24
```

[coredump](https://get.paskal-dev.com/core-envoy-6-101-0-20-1645993355)

```
envoy version: a9d72603c68da3a10a1c0d021d01c7877e6f2a30/1.21.0/Clean/RELEASE/BoringSS
```

we turn of TTL on xDS messages and envoy stops crashing

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.