open-telemetry / open-telemetry/opentelemetry-cpp-contrib

Coredumps when the OpenTelemetry collector host is unavailable.

Open
#395 0 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
153
Forks
184
Avg merge
2d 9h
Merged PRs (30d)
26

Description

Describe your environment
OpenTelemetry sdk: v1.8.1
Nginx: v1.21.6
Ingress-Nginx: v1.8.4
k8s: 1.24.6

Steps to reproduce

  • HIghload Nginx.
  • The collector's OpenTelemetry host is unavailable.

What is the expected behavior:
The absence of Coredumps when the OpenTelemetry collector host is unavailable.

What is the actual behavior:
Coredumps files are created

Additional context
Example Coredump:

(gdb) backtrace full
#0  0x00007fb00cc99b8a in opentelemetry::v1::exporter::otlp::OtlpGrpcExporter::Export(opentelemetry::v1::nostd::span<std::unique_ptr<opentelemetry::v1::sdk::trace::Recordable, std::default_delete<opentelemetry::v1::sdk::trace::Recordable> >, 18446744073709551615ul> const&) ()
   from /modules_mount/etc/nginx/modules/otel/otel_ngx_module.so
No symbol table info available.
#1  0x00007fb00cc91321 in opentelemetry::v1::sdk::trace::BatchSpanProcessor::Export() () from /modules_mount/etc/nginx/modules/otel/otel_ngx_module.so
No symbol table info available.
#2  0x00007fb00cc909c1 in opentelemetry::v1::sdk::trace::BatchSpanProcessor::DrainQueue() () from /modules_mount/etc/nginx/modules/otel/otel_ngx_module.so
No symbol table info available.
#3  0x00007fb00cc90b78 in opentelemetry::v1::sdk::trace::BatchSpanProcessor::DoBackgroundWork() () from /modules_mount/etc/nginx/modules/otel/otel_ngx_module.so
No symbol table info available.
#4  0x00007fb00c8711e3 in ?? () from /usr/lib/libstdc++.so.6
No symbol table info available.
#5  0x00007fb00de04bc2 in ?? () from /lib/ld-musl-x86_64.so.1
No symbol table info available.
#6  0x0000000000000000 in ?? ()
No symbol table info available.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the high-load Nginx scenario with the OpenTelemetry collector host unavailable, then inspect OtlpGrpcExporter::Export and the BatchSpanProcessor::Export, DrainQueue, and DoBackgroundWork entry points shown in the backtrace. Done means the collector outage no longer creates coredump files.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, kubernetes, nginx
Domain
infrastructure, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.