open-telemetry / open-telemetry/opentelemetry-cpp-contrib
Getting UNAVAILABLE: Socket closed while connecting to Elastic APM Server
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 153
- Forks
- 184
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 26
Description
We are trying to perform a POC to get nginx traces to Elastic APM Server using the grpc protocol.
otel agent -- > APM server is not working and giving the error UNAVAILABLE: Socket closed
although the flow otel agent --> otel collector --> APM server is working
Both the agent and otel collector are deployed on same K8's env.
Otel collector configuration looks like based on example https://github.com/esigo/nginx-example
collector.yaml
exporters:
otlp:
endpoint: "<apm server url>:8200"
headers:
# Elastic APM Server API key
Authorization: "Bearer <token>"
tls:
insecure: false
insecure_skip_verify: true
Steps to reproduce
Try to install ingress-nginx and point the endpoint to APM server
helm upgrade --install ingress-nginx ingress-nginx/ingress-nginx
--namespace ingress-nginx --create-namespace --set controller.opentelemetry.enabled=true
--set controller.service.type=NodePort -f ingress/controller-config.yml
controller-config.yaml
controller:
opentelemetry:
enabled: "true"
config:
log-level: "debug"
enable-opentelemetry: "true"
opentelemetry-operation-name: "HTTP $request_method $service_name $uri"
opentelemetry-trust-incoming-span: "true"
otlp-collector-host: "<apm server url>"
otlp-collector-port: "8200"
otel-exporter-otlp-headers: "Authorization=Bearer <token>"
otel-max-queuesize: "2048"
otel-schedule-delay-millis: "5000"
otel-max-export-batch-size: "512"
otel-service-name: "nginx-proxy" # Opentelemetry resource name
otel-sampler: "AlwaysOn" # Also: AlwaysOff, TraceIdRatioBased
otel-sampler-ratio: "1.0"
otel-sampler-parent-based: "true"
What is the expected behavior?
The Agent should be able to send the traces to APM Server as APM Server also support the grpc protocol
What is the actual behavior?
We are getting this error
[Error] File: /tmp/build/opentelemetry-cpp-v1.8.1/exporters/otlp/src/otlp_grpc_exporter.cc:67[OTLP TRACE GRPC Exporter] Export() failed with status_code: "UNAVAILABLE" error_message: "failed to connect to all addresses; last error: UNAVAILABLE: Socket closed"
Additional context
Add any other context about the problem here.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the ingress-nginx controller configuration and collector.yaml, then compare the direct OTLP gRPC path to the working collector path. Review the reported otlp_grpc_exporter.cc error and the APM Server endpoint, TLS, and authorization settings. Done means identifying why the direct connection closes and documenting a configuration that sends traces successfully.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, grpc, kubernetes
- Domain
- infrastructure, observability-sre
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100