opentelemetry directives unknown

Describe your environment

Running nginx in a kubernetes pod to serve static content and proxy dynamic requests.

Steps to reproduce

Nginx pod was build with this dockerfile:

# Take an upstream docker image as an argument
# build the static directory there

# SRC_IMG will be the `web` container, with the assets collected in /app/static-collected
ARG SRC_IMG

FROM ${SRC_IMG} AS static_source

# When updating nginx also make sure to check otel module compatibility matrix and update otel version accordingly
FROM nginx:1.23.1
ADD https://github.com/open-telemetry/opentelemetry-cpp-contrib/releases/download/webserver%2Fv1.0.3/opentelemetry-webserver-sdk-x64-linux.tgz /opt
RUN cd /opt; tar xvfz opentelemetry-webserver-sdk-x64-linux.tgz
RUN cd /opt/opentelemetry-webserver-sdk; ./install.sh
ENV LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/opentelemetry-webserver-sdk/sdk_lib/lib
RUN mkdir /app/
COPY --from=static_source /app/static-collected /app/static/

adding in static content is probably not necessary to reproduce.

nginx.conf, otel-nginx.toml and also opentelemetry_module.conf are loaded into the pod via configmap:

data:
  nginx.template.conf: |-
    load_module /opt/opentelemetry-webserver-sdk/WebServerModule/Nginx/${NGINX_VERSION}/ngx_http_opentelemetry_module.so;

    user www-data;
    worker_processes 8;
    error_log  /var/log/nginx/error.log warn;
    pid        /var/run/nginx.pid;

    # https://serverfault.com/questions/787919/optimal-value-for-nginx-worker-connections
    events {
      worker_connections 10240;
    }

    http {
      opentelemetry_config /conf/otel-nginx.toml;
      
      upstream varnish_upstream {
        server ${VARNISH_HOST}:8040;
        keepalive 32;
      }

      server {
        listen 80 default_server;
        listen [::]:80;
        server_name _; 

        location / {
            proxy_send_timeout  300;
            proxy_read_timeout  300;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto https;
            proxy_set_header X-Forwarded-Port 443;
            proxy_pass http://varnish_upstream;
        }

      } # server
    } # http
  opentelemetry_module.conf: |-
    NginxModuleEnabled ON;
    NginxModuleOtelSpanExporter otlp;
    NginxModuleOtelExporterEndpoint http://otel-collector-collector.monitoring:4317;
    NginxModuleServiceName nginx-instrumentation;
    NginxModuleServiceNamespace default;
    NginxModuleServiceInstanceId nginx-instrumentation-test;
    NginxModuleResolveBackends ON;
    NginxModuleTraceAsError OFF;
  otel-nginx.toml: |-
    exporter = "otlp"
    processor = "batch"

    [exporters.otlp]

    host = "otel-collector-collector.monitoring.svc.cluster.local"
    port = 4317

    [processors.batch]
    max_queue_size = 2048
    schedule_delay_millis = 5000
    max_export_batch_size = 512

    [service]
    name = instrumentation-nginx

and the nginx.conf file is templated with the varnish and nginx version values before nginx starts.

What is the expected behavior?
Nginx starts correctly and sends traces to the otel collector

What is the actual behavior?

nginx crashes with the message:

nginx: [emerg] unknown directive "opentelemetry_config" in /nginx.conf:14

Additional context

I have also tried using the instructions in the opentelemetry blog which configure the module using include /etc/nginx/config.d/*.conf instead of the opentelemetry_config directive. In that case nginx starts up and I see the error I describe in !255. Traces appear in Jaeger, but seemingly without propagation configured. If I try add opentelemetry directives, such as opentelemetry_propogation or opentelemetry_trust_upstream_traces, nginx crashes as described above

Start with the Dockerfile and the templated nginx.conf, then inspect otel-nginx.toml and opentelemetry_module.conf alongside the module compatibility matrix. Compare the loaded module and supported directives with the blog configuration and the reported nginx errors. Done means nginx starts with the intended configuration and traces retain the expected propagation behavior.