open-telemetry / open-telemetry/opentelemetry-cpp-contrib

otel-webserver-module Fails to run on Redhat 7

Open
#337 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug Webserver
Dominant language
Python
Stars
153
Forks
184
Avg merge
2d 9h
Merged PRs (30d)
26

Description

Describe your environment

NAME="Red Hat Enterprise Linux Server"
VERSION="7.9 (Maipo)"
[root@ip-10-56-1-18 ld.so.conf.d]# nginx -V
nginx version: nginx/1.22.0
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC)
built with OpenSSL 1.0.2k-fips  26 Jan 2017
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -fPIC' --with-ld-opt='-Wl,-z,relro -Wl,-z,now -pie'
[root@ip-10-56-1-18 ld.so.conf.d]# nginx -t
nginx: [error] mod_opentelemetry: ngx_http_opentelemetry_init: Starting Opentelemetry Modlue init
nginx: [error] mod_opentelemetry: ngx_http_opentelemetry_init: Registering handlers for modules in different phases
nginx: [error] mod_opentelemetry: ngx_http_opentelemetry_init: Opentelemetry Modlue init completed !
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
[root@ip-10-56-1-18 ld.so.conf.d]# ldconfig -v

/opt/opentelemetry-webserver-sdk/sdk_lib/lib:
        libopentelemetry_exporter_otlp_grpc.so -> libopentelemetry_exporter_otlp_grpc.so
        libopentelemetry_webserver_sdk.so -> libopentelemetry_webserver_sdk.so
        libopentelemetry_common.so -> libopentelemetry_common.so
        libopentelemetry_trace.so -> libopentelemetry_trace.so
        libopentelemetry_exporter_ostream_span.so -> libopentelemetry_exporter_ostream_span.so
        libopentelemetry_otlp_recordable.so -> libopentelemetry_otlp_recordable.so
        libopentelemetry_resources.so -> libopentelemetry_resources.so

partial Config

load_module /opt/opentelemetry-webserver-sdk/WebServerModule/Nginx/1.22.0/ngx_http_opentelemetry_module.so;

http {
  # Nginx will handle gzip compression of responses from the app server
  gzip on;
  gzip_proxied any;
  gzip_types text/plain application/json;
  gzip_min_length 1000;
  include /etc/nginx/conf.d/opentelemetry_module.conf;
# configuration file /etc/nginx/conf.d/opentelemetry_module.conf:
NginxModuleEnabled ON;
NginxModuleOtelSpanExporter otlp;
NginxModuleOtelExporterEndpoint tempo.jaeger.dev.company.network:4317;
NginxModuleServiceName RBCS;
NginxModuleServiceNamespace DEV;
NginxModuleServiceInstanceId jaeger;
NginxModuleResolveBackends ON;
NginxModuleTraceAsError On;

Steps to reproduce

  1. Boot up RHEL 7 AMI
  2. Install nginx from the nginx stable repo
  3. Download/install the latest otel-webserver-module release
  4. create /etc/ld.so.conf.d/nginx.conf and add /opt/opentelemetry-webserver-sdk/sdk_lib/lib to the file
  5. Create /etc/nginx/conf.d/opentelemetry_module.conf
  6. Start NGINX

What is the expected behavior?
I would expect to see traces, we use the same config, on debian docker images and have no problems.

What is the actual behavior?

2023/11/13 19:50:26 [error] 4480#4480: mod_opentelemetry: ngx_http_opentelemetry_init: Starting Opentelemetry Modlue init
2023/11/13 19:50:26 [error] 4480#4480: mod_opentelemetry: ngx_http_opentelemetry_init: Registering handlers for modules in different phases
2023/11/13 19:50:26 [error] 4480#4480: mod_opentelemetry: ngx_http_opentelemetry_init: Opentelemetry Modlue init completed !
2023/11/13 19:50:26 [error] 4482#4482: mod_opentelemetry: ngx_http_opentelemetry_init_worker: Initializing Nginx Worker for process with PID: 4482
2023/11/13 19:50:26 [error] 4483#4483: mod_opentelemetry: ngx_http_opentelemetry_init_worker: Initializing Nginx Worker for process with PID: 4483
Error: virtual OTEL_SDK_STATUS_CODE otel::core::ApiUtils::init_boilerplate(): Invalid logging config file: "/opt/opentelemetry-webserver-sdk/conf/appdynamics_sdk_log4cxx.xml"
Error: OTEL_SDK_STATUS_CODE otel::core::WSAgent::init(OTEL_SDK_ENV_RECORD*, unsigned int): Error Code: 8
2023/11/13 19:50:28 [error] 4482#4482: *2 mod_opentelemetry: ngx_initialize_opentelemetry: Agent Core Init failed, result code is 8, client: 10.56.1.82, server: , request: "GET /health HTTP/1.1", host: "10.56.1.82:443"
2023/11/13 19:50:28 [error] 4482#4482: *2 mod_opentelemetry: startMonitoringRequest: Opentelemetry Agent Core did not get initialized, client: 10.56.1.82, server: , request: "GET /health HTTP/1.1", host: "10.56.1.82:443"
2023/11/13 19:50:28 [error] 4482#4482: *2 mod_opentelemetry: traceConfig:  Config { :(Enabled="1")(OtelExporterEndpoint="tempo.jaeger.dev.company.network:4317")(OtelSslEnabled="0")(OtelSslCertificatePath="")(OtelSpanExporter="otlp")(OtelSpanProcessor="")(OtelSampler="")(ServiceNamespace="DEV")(ServiceName="RBCS")(ServiceInstanceId="jaeger")(OtelMaxQueueSize="2048")(OtelScheduledDelayMillis="5000")(OtelExportTimeoutMillis="30000")(OtelMaxExportBatchSize="512")(ResolveBackends="1")(TraceAsError="1")(ReportAllInstrumentedModules="0")(MaskCookie="0")(MaskSmUser="0")(SegmentType="First")(SegmentParameter="2") }, client: 10.56.1.82, server: , request: "GET /health HTTP/1.1", host: "10.56.1.82:443"

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the failure on RHEL 7 using the nginx version, module path, /etc/ld.so.conf.d/nginx.conf, and opentelemetry_module.conf shown in the report. Start with the appdynamics_sdk_log4cxx.xml error during worker initialization and compare the module startup behavior with the reported Debian setup. Done means the agent initializes without error and requests produce traces.

Written by the indexing model from the issue text.

Assessment

Tech stack
linux, nginx
Domain
observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.