open-telemetry / open-telemetry/opentelemetry-cpp
Crash in OTLP HTTP Export: WITH_ASYNC_EXPORT_PREVIEW=ON
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 1.4k
- Forks
- 632
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 75
Description
Describe your environment
Build and running on Ubuntu 24.04 ARM64 (VM inside M1 Macbook)
cmake .. -DBUILD_SHARED_LIBS=ON -DWITH_OTLP_HTTP=ON -DWITH_OTLP_FILE=ON
-DCMAKE_INSTALL_RPATH_USE_LINK_PATH=ON -DBUILD_TESTING=OFF -DWITH_ASYNC_EXPORT_PREVIEW=ON
Protobuf Version: 3.21.12
OTEL Version: 1.16.1
Steps to reproduce
main.cpp.txt
tracer_common.h.txt
Backtrace
#0 __pthread_kill_implementation (threadid=281474840629312, signo=signo@entry=6, no_tid=no_tid@entry=0) at ./nptl/pthread_kill.c:44
#1 0x0000fffff78c7690 in __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78
#2 0x0000fffff787cb3c in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#3 0x0000fffff7867e00 in __GI_abort () at ./stdlib/abort.c:79
#4 0x0000fffff78babf0 in __libc_message_impl (fmt=fmt@entry=0xfffff79a4a60 "%s\n") at ../sysdeps/posix/libc_fatal.c:132
#5 0x0000fffff78d212c in malloc_printerr (str=str@entry=0xfffff799fe10 "double free or corruption (!prev)") at ./malloc/malloc.c:5772
#6 0x0000fffff78d4248 in _int_free_merge_chunk (av=av@entry=0xfffff79f0a50 <main_arena>, p=p@entry=0xaaaaaab1ab40, size=size@entry=496) at ./malloc/malloc.c:4679
#7 0x0000fffff78d43f8 in _int_free (av=0xfffff79f0a50 <main_arena>, p=p@entry=0xaaaaaab1ab40, have_lock=<optimized out>, have_lock@entry=0) at ./malloc/malloc.c:4646
#8 0x0000fffff78d6fa8 in __GI___libc_free (mem=<optimized out>) at ./malloc/malloc.c:3398
#9 0x0000fffff7e07320 in opentelemetry::v1::exporter::otlp::OtlpHttpExporter::~OtlpHttpExporter() () from /usr/local/lib/libopentelemetry_exporter_otlp_http.so
#10 0x0000aaaaaaabce78 in std::default_delete<opentelemetry::v1::sdk::trace::SpanExporter>::operator() (this=0xaaaaaab1bba8, __ptr=0xaaaaaab1ab50) at /usr/include/c++/13/bits/unique_ptr.h:99
#11 0x0000aaaaaaabb53c in std::unique_ptr<opentelemetry::v1::sdk::trace::SpanExporter, std::default_delete<opentelemetry::v1::sdk::trace::SpanExporter> >::~unique_ptr (this=0xaaaaaab1bba8 = {...}, __in_chrg=<optimized out>) at /usr/include/c++/13/bits/unique_ptr.h:404
#12 0x0000fffff7f6d460 in opentelemetry::v1::sdk::trace::BatchSpanProcessor::~BatchSpanProcessor() () from /usr/local/lib/libopentelemetry_trace.so
#13 0x0000fffff7f6d488 in opentelemetry::v1::sdk::trace::BatchSpanProcessor::~BatchSpanProcessor() () from /usr/local/lib/libopentelemetry_trace.so
warning: RTTI symbol not found for class 'opentelemetry::v1::sdk::trace::BatchSpanProcessor'
#14 0x0000aaaaaaabcefc in std::default_delete<opentelemetry::v1::sdk::trace::SpanProcessor>::operator() (this=0xaaaaaab2baa0, __ptr=0xaaaaaab1bba0) at /usr/include/c++/13/bits/unique_ptr.h:99
#15 0x0000aaaaaaabb5b8 in std::unique_ptr<opentelemetry::v1::sdk::trace::SpanProcessor, std::default_delete<opentelemetry::v1::sdk::trace::SpanProcessor> >::~unique_ptr (this=0xaaaaaab2baa0 = {...}, __in_chrg=<optimized out>) at /usr/include/c++/13/bits/unique_ptr.h:404
#16 0x0000fffff7f379fc in opentelemetry::v1::sdk::trace::MultiSpanProcessor::ProcessorNode::~ProcessorNode() () from /usr/local/lib/libopentelemetry_trace.so
#17 0x0000fffff7f37aa8 in opentelemetry::v1::sdk::trace::MultiSpanProcessor::Cleanup() () from /usr/local/lib/libopentelemetry_trace.so
#18 0x0000fffff7f37900 in opentelemetry::v1::sdk::trace::MultiSpanProcessor::~MultiSpanProcessor() () from /usr/local/lib/libopentelemetry_trace.so
#19 0x0000fffff7f37928 in opentelemetry::v1::sdk::trace::MultiSpanProcessor::~MultiSpanProcessor() () from /usr/local/lib/libopentelemetry_trace.so
warning: RTTI symbol not found for class 'opentelemetry::v1::sdk::trace::MultiSpanProcessor'
#20 0x0000aaaaaaabcefc in std::default_delete<opentelemetry::v1::sdk::trace::SpanProcessor>::operator() (this=0xaaaaaab152a8, __ptr=0xaaaaaab1cf90) at /usr/include/c++/13/bits/unique_ptr.h:99
#21 0x0000aaaaaaabb5b8 in std::unique_ptr<opentelemetry::v1::sdk::trace::SpanProcessor, std::default_delete<opentelemetry::v1::sdk::trace::SpanProcessor> >::~unique_ptr (this=0xaaaaaab152a8 = {...}, __in_chrg=<optimized out>) at /usr/include/c++/13/bits/unique_ptr.h:404
#22 0x0000fffff7f44d78 in opentelemetry::v1::sdk::trace::TracerContext::~TracerContext() () from /usr/local/lib/libopentelemetry_trace.so
#23 0x0000fffff7f44dbc in opentelemetry::v1::sdk::trace::TracerContext::~TracerContext() () from /usr/local/lib/libopentelemetry_trace.so
#24 0x0000aaaaaaabd118 in std::default_delete<opentelemetry::v1::sdk::trace::TracerContext>::operator() (this=0xaaaaaab27c90, __ptr=0xaaaaaab15230) at /usr/include/c++/13/bits/unique_ptr.h:99
#25 0x0000fffff7f6032c in std::_Sp_counted_deleter<opentelemetry::v1::sdk::trace::TracerContext*, std::default_delete<opentelemetry::v1::sdk::trace::TracerContext>, std::allocator<void>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /usr/local/lib/libopentelemetry_trace.so
warning: RTTI symbol not found for class 'std::_Sp_counted_deleter<opentelemetry::v1::sdk::trace::TracerContext*, std::default_delete<opentelemetry::v1::sdk::trace::TracerContext>, std::allocator<void>, (__gnu_cxx::_Lock_policy)2>'
#26 0x0000aaaaaaab7ca4 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0xaaaaaab27c80) at /usr/include/c++/13/bits/shared_ptr_base.h:346
#27 0x0000aaaaaaabb014 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count (this=0xaaaaaab2b630, __in_chrg=<optimized out>) at /usr/include/c++/13/bits/shared_ptr_base.h:1071
#28 0x0000fffff7f4843c in std::__shared_ptr<opentelemetry::v1::sdk::trace::TracerContext, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() () from /usr/local/lib/libopentelemetry_trace.so
#29 0x0000fffff7f4845c in std::shared_ptr<opentelemetry::v1::sdk::trace::TracerContext>::~shared_ptr() () from /usr/local/lib/libopentelemetry_trace.so
#30 0x0000fffff7f54084 in opentelemetry::v1::sdk::trace::Tracer::~Tracer() () from /usr/local/lib/libopentelemetry_trace.so
#31 0x0000fffff7f540c4 in opentelemetry::v1::sdk::trace::Tracer::~Tracer() () from /usr/local/lib/libopentelemetry_trace.so
#32 0x0000fffff7f6008c in std::_Sp_counted_ptr<opentelemetry::v1::sdk::trace::Tracer*, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /usr/local/lib/libopentelemetry_trace.so
warning: RTTI symbol not found for class 'std::_Sp_counted_ptr<opentelemetry::v1::sdk::trace::Tracer*, (__gnu_cxx::_Lock_policy)2>'
#33 0x0000aaaaaaabba14 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release_last_use (this=0xaaaaaab2b810) at /usr/include/c++/13/bits/shared_ptr_base.h:175
warning: RTTI symbol not found for class 'std::_Sp_counted_ptr<opentelemetry::v1::sdk::trace::Tracer*, (__gnu_cxx::_Lock_policy)2>'
#34 0x0000aaaaaaaba070 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release_last_use_cold (this=0xaaaaaab2b810) at /usr/include/c++/13/bits/shared_ptr_base.h:199
warning: RTTI symbol not found for class 'std::_Sp_counted_ptr<opentelemetry::v1::sdk::trace::Tracer*, (__gnu_cxx::_Lock_policy)2>'
#35 0x0000aaaaaaab7d74 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0xaaaaaab2b810) at /usr/include/c++/13/bits/shared_ptr_base.h:353
#36 0x0000aaaaaaabb014 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count (this=0xaaaaaab2ae18, __in_chrg=<optimized out>) at /usr/include/c++/13/bits/shared_ptr_base.h:1071
#37 0x0000fffff7f485e4 in std::__shared_ptr<opentelemetry::v1::sdk::trace::Tracer, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() () from /usr/local/lib/libopentelemetry_trace.so
#38 0x0000fffff7f48604 in std::shared_ptr<opentelemetry::v1::sdk::trace::Tracer>::~shared_ptr() () from /usr/local/lib/libopentelemetry_trace.so
#39 0x0000fffff7f56d08 in void std::_Destroy<std::shared_ptr<opentelemetry::v1::sdk::trace::Tracer> >(std::shared_ptr<opentelemetry::v1::sdk::trace::Tracer>*) () from /usr/local/lib/libopentelemetry_trace.so
#40 0x0000fffff7f53914 in void std::_Destroy_aux<false>::__destroy<std::shared_ptr<opentelemetry::v1::sdk::trace::Tracer>*>(std::shared_ptr<opentelemetry::v1::sdk::trace::Tracer>*, std::shared_ptr<opentelemetry::v1::sdk::trace::Tracer>*) () from /usr/local/lib/libopentelemetry_trace.so
#41 0x0000fffff7f50d98 in void std::_Destroy<std::shared_ptr<opentelemetry::v1::sdk::trace::Tracer>*>(std::shared_ptr<opentelemetry::v1::sdk::trace::Tracer>*, std::shared_ptr<opentelemetry::v1::sdk::trace::Tracer>*) () from /usr/local/lib/libopentelemetry_trace.so
#42 0x0000fffff7f4a100 in std::vector<std::shared_ptr<opentelemetry::v1::sdk::trace::Tracer>, std::allocator<std::shared_ptr<opentelemetry::v1::sdk::trace::Tracer> > >::~vector() () from /usr/local/lib/libopentelemetry_trace.so
#43 0x0000fffff7f45d5c in opentelemetry::v1::sdk::trace::TracerProvider::~TracerProvider() () from /usr/local/lib/libopentelemetry_trace.so
#44 0x0000fffff7f45d88 in opentelemetry::v1::sdk::trace::TracerProvider::~TracerProvider() () from /usr/local/lib/libopentelemetry_trace.so
#45 0x0000aaaaaaabd1bc in std::default_delete<opentelemetry::v1::sdk::trace::TracerProvider>::operator() (this=0xaaaaaab27cd0, __ptr=0xaaaaaab296c0) at /usr/include/c++/13/bits/unique_ptr.h:99
#46 0x0000aaaaaaac0818 in std::_Sp_counted_deleter<opentelemetry::v1::sdk::trace::TracerProvider*, std::default_delete<opentelemetry::v1::sdk::trace::TracerProvider>, std::allocator<void>, (__gnu_cxx::_Lock_policy)2>::_M_dispose (this=0xaaaaaab27cc0) at /usr/include/c++/13/bits/shared_ptr_base.h:527
#47 0x0000aaaaaaab7ca4 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0xaaaaaab27cc0) at /usr/include/c++/13/bits/shared_ptr_base.h:346
#48 0x0000aaaaaaabb014 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count (this=0xaaaaaaae0098 <opentelemetry::v1::trace::Provider::GetProvider()::provider+16>, __in_chrg=<optimized out>) at /usr/include/c++/13/bits/shared_ptr_base.h:1071
#49 0x0000aaaaaaab9ef8 in std::__shared_ptr<opentelemetry::v1::trace::TracerProvider, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr (this=0xaaaaaaae0090 <opentelemetry::v1::trace::Provider::GetProvider()::provider+8>, __in_chrg=<optimized out>) at /usr/include/c++/13/bits/shared_ptr_base.h:1524
#50 0x0000aaaaaaab9f18 in std::shared_ptr<opentelemetry::v1::trace::TracerProvider>::~shared_ptr (this=0xaaaaaaae0090 <opentelemetry::v1::trace::Provider::GetProvider()::provider+8> = {...}, __in_chrg=<optimized out>) at /usr/include/c++/13/bits/shared_ptr.h:175
#51 0x0000aaaaaaabccc4 in opentelemetry::v1::nostd::shared_ptr<opentelemetry::v1::trace::TracerProvider>::shared_ptr_wrapper::~shared_ptr_wrapper (this=0xaaaaaaae0088 <opentelemetry::v1::trace::Provider::GetProvider()::provider>, __in_chrg=<optimized out>) at /usr/local/include/opentelemetry/nostd/shared_ptr.h:50
#52 0x0000aaaaaaabb3a8 in opentelemetry::v1::nostd::shared_ptr<opentelemetry::v1::trace::TracerProvider>::~shared_ptr (this=0xaaaaaaae0088 <opentelemetry::v1::trace::Provider::GetProvider()::provider>, __in_chrg=<optimized out>) at /usr/local/include/opentelemetry/nostd/shared_ptr.h:118
#53 0x0000fffff787f228 in __run_exit_handlers (status=0, listp=0xfffff79f0670 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at ./stdlib/exit.c:108
#54 0x0000fffff787f30c in __GI_exit (status=<optimized out>) at ./stdlib/exit.c:138
#55 0x0000fffff78684c8 in __libc_start_call_main (main=main@entry=0xaaaaaaab7958 <main()>, argc=argc@entry=1, argv=argv@entry=0xfffffffff9d8) at ../sysdeps/nptl/libc_start_call_main.h:74
#56 0x0000fffff7868598 in __libc_start_main_impl (main=0xaaaaaaab7958 <main()>, argc=1, argv=0xfffffffff9d8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=<optimized out>) at ../csu/libc-start.c:360
#57 0x0000aaaaaaab71f0 in _start ()
Additional context
This happens during the destructor call of OtlpHttpExporter and when Async export is enabled.
Didn't understood why this is happening. Need some help.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the attached main.cpp.txt and tracer_common.h.txt, then reproduce the crash using the shown CMake configuration with WITH_ASYNC_EXPORT_PREVIEW=ON. Inspect the OtlpHttpExporter destructor and the async exporter shutdown path indicated by the backtrace. Done means the reproducer exits without double-free or heap-corruption errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- observability-sre
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100