open-telemetry / open-telemetry/opentelemetry-cpp-contrib

Getting UNAVAILABLE: Socket closed while connecting to Elastic APM Server

Open
#324 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
153
Forks
184
Avg merge
2d 9h
Merged PRs (30d)
26

Description

We are trying to perform a POC to get nginx traces to Elastic APM Server using the grpc protocol.

otel agent -- > APM server is not working and giving the error UNAVAILABLE: Socket closed

although the flow otel agent --> otel collector --> APM server is working

Both the agent and otel collector are deployed on same K8's env.

Otel collector configuration looks like based on example https://github.com/esigo/nginx-example

collector.yaml

    exporters:
      otlp:
        endpoint: "<apm server url>:8200"
        headers:
          # Elastic APM Server API key
          Authorization: "Bearer <token>"
        tls:
          insecure: false
          insecure_skip_verify: true

Steps to reproduce
Try to install ingress-nginx and point the endpoint to APM server

helm upgrade --install ingress-nginx ingress-nginx/ingress-nginx
--namespace ingress-nginx --create-namespace --set controller.opentelemetry.enabled=true
--set controller.service.type=NodePort -f ingress/controller-config.yml

controller-config.yaml


controller:
  opentelemetry:
    enabled: "true"
  config:
    log-level: "debug"
    enable-opentelemetry: "true"
    opentelemetry-operation-name: "HTTP $request_method $service_name $uri"
    opentelemetry-trust-incoming-span: "true"
    otlp-collector-host: "<apm server url>"
    otlp-collector-port: "8200"
    otel-exporter-otlp-headers: "Authorization=Bearer <token>"
    otel-max-queuesize: "2048"
    otel-schedule-delay-millis: "5000"
    otel-max-export-batch-size: "512"
    otel-service-name: "nginx-proxy" # Opentelemetry resource name
    otel-sampler: "AlwaysOn" # Also: AlwaysOff, TraceIdRatioBased
    otel-sampler-ratio: "1.0"
    otel-sampler-parent-based: "true"


What is the expected behavior?
The Agent should be able to send the traces to APM Server as APM Server also support the grpc protocol

What is the actual behavior?
We are getting this error

[Error] File: /tmp/build/opentelemetry-cpp-v1.8.1/exporters/otlp/src/otlp_grpc_exporter.cc:67[OTLP TRACE GRPC Exporter] Export() failed with status_code: "UNAVAILABLE" error_message: "failed to connect to all addresses; last error: UNAVAILABLE: Socket closed"

Additional context
Add any other context about the problem here.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the ingress-nginx controller configuration and collector.yaml, then compare the direct OTLP gRPC path to the working collector path. Review the reported otlp_grpc_exporter.cc error and the APM Server endpoint, TLS, and authorization settings. Done means identifying why the direct connection closes and documenting a configuration that sends traces successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, grpc, kubernetes
Domain
infrastructure, observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.