open-telemetry / open-telemetry/opentelemetry-cpp-contrib
otel-webserver-module Fails to run on Redhat 7
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 153
- Forks
- 184
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 26
Description
Describe your environment
NAME="Red Hat Enterprise Linux Server"
VERSION="7.9 (Maipo)"
[root@ip-10-56-1-18 ld.so.conf.d]# nginx -V
nginx version: nginx/1.22.0
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC)
built with OpenSSL 1.0.2k-fips 26 Jan 2017
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -fPIC' --with-ld-opt='-Wl,-z,relro -Wl,-z,now -pie'
[root@ip-10-56-1-18 ld.so.conf.d]# nginx -t
nginx: [error] mod_opentelemetry: ngx_http_opentelemetry_init: Starting Opentelemetry Modlue init
nginx: [error] mod_opentelemetry: ngx_http_opentelemetry_init: Registering handlers for modules in different phases
nginx: [error] mod_opentelemetry: ngx_http_opentelemetry_init: Opentelemetry Modlue init completed !
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
[root@ip-10-56-1-18 ld.so.conf.d]# ldconfig -v
/opt/opentelemetry-webserver-sdk/sdk_lib/lib:
libopentelemetry_exporter_otlp_grpc.so -> libopentelemetry_exporter_otlp_grpc.so
libopentelemetry_webserver_sdk.so -> libopentelemetry_webserver_sdk.so
libopentelemetry_common.so -> libopentelemetry_common.so
libopentelemetry_trace.so -> libopentelemetry_trace.so
libopentelemetry_exporter_ostream_span.so -> libopentelemetry_exporter_ostream_span.so
libopentelemetry_otlp_recordable.so -> libopentelemetry_otlp_recordable.so
libopentelemetry_resources.so -> libopentelemetry_resources.so
partial Config
load_module /opt/opentelemetry-webserver-sdk/WebServerModule/Nginx/1.22.0/ngx_http_opentelemetry_module.so;
http {
# Nginx will handle gzip compression of responses from the app server
gzip on;
gzip_proxied any;
gzip_types text/plain application/json;
gzip_min_length 1000;
include /etc/nginx/conf.d/opentelemetry_module.conf;
# configuration file /etc/nginx/conf.d/opentelemetry_module.conf:
NginxModuleEnabled ON;
NginxModuleOtelSpanExporter otlp;
NginxModuleOtelExporterEndpoint tempo.jaeger.dev.company.network:4317;
NginxModuleServiceName RBCS;
NginxModuleServiceNamespace DEV;
NginxModuleServiceInstanceId jaeger;
NginxModuleResolveBackends ON;
NginxModuleTraceAsError On;
Steps to reproduce
- Boot up RHEL 7 AMI
- Install nginx from the nginx stable repo
- Download/install the latest otel-webserver-module release
- create
/etc/ld.so.conf.d/nginx.confand add/opt/opentelemetry-webserver-sdk/sdk_lib/libto the file - Create
/etc/nginx/conf.d/opentelemetry_module.conf - Start NGINX
What is the expected behavior?
I would expect to see traces, we use the same config, on debian docker images and have no problems.
What is the actual behavior?
2023/11/13 19:50:26 [error] 4480#4480: mod_opentelemetry: ngx_http_opentelemetry_init: Starting Opentelemetry Modlue init
2023/11/13 19:50:26 [error] 4480#4480: mod_opentelemetry: ngx_http_opentelemetry_init: Registering handlers for modules in different phases
2023/11/13 19:50:26 [error] 4480#4480: mod_opentelemetry: ngx_http_opentelemetry_init: Opentelemetry Modlue init completed !
2023/11/13 19:50:26 [error] 4482#4482: mod_opentelemetry: ngx_http_opentelemetry_init_worker: Initializing Nginx Worker for process with PID: 4482
2023/11/13 19:50:26 [error] 4483#4483: mod_opentelemetry: ngx_http_opentelemetry_init_worker: Initializing Nginx Worker for process with PID: 4483
Error: virtual OTEL_SDK_STATUS_CODE otel::core::ApiUtils::init_boilerplate(): Invalid logging config file: "/opt/opentelemetry-webserver-sdk/conf/appdynamics_sdk_log4cxx.xml"
Error: OTEL_SDK_STATUS_CODE otel::core::WSAgent::init(OTEL_SDK_ENV_RECORD*, unsigned int): Error Code: 8
2023/11/13 19:50:28 [error] 4482#4482: *2 mod_opentelemetry: ngx_initialize_opentelemetry: Agent Core Init failed, result code is 8, client: 10.56.1.82, server: , request: "GET /health HTTP/1.1", host: "10.56.1.82:443"
2023/11/13 19:50:28 [error] 4482#4482: *2 mod_opentelemetry: startMonitoringRequest: Opentelemetry Agent Core did not get initialized, client: 10.56.1.82, server: , request: "GET /health HTTP/1.1", host: "10.56.1.82:443"
2023/11/13 19:50:28 [error] 4482#4482: *2 mod_opentelemetry: traceConfig: Config { :(Enabled="1")(OtelExporterEndpoint="tempo.jaeger.dev.company.network:4317")(OtelSslEnabled="0")(OtelSslCertificatePath="")(OtelSpanExporter="otlp")(OtelSpanProcessor="")(OtelSampler="")(ServiceNamespace="DEV")(ServiceName="RBCS")(ServiceInstanceId="jaeger")(OtelMaxQueueSize="2048")(OtelScheduledDelayMillis="5000")(OtelExportTimeoutMillis="30000")(OtelMaxExportBatchSize="512")(ResolveBackends="1")(TraceAsError="1")(ReportAllInstrumentedModules="0")(MaskCookie="0")(MaskSmUser="0")(SegmentType="First")(SegmentParameter="2") }, client: 10.56.1.82, server: , request: "GET /health HTTP/1.1", host: "10.56.1.82:443"
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the failure on RHEL 7 using the nginx version, module path, /etc/ld.so.conf.d/nginx.conf, and opentelemetry_module.conf shown in the report. Start with the appdynamics_sdk_log4cxx.xml error during worker initialization and compare the module startup behavior with the reported Debian setup. Done means the agent initializes without error and requests produce traces.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- linux, nginx
- Domain
- observability-sre
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100