open-telemetry / open-telemetry/opentelemetry-python

Distros cannot override the default propagators via OTEL_PROPAGATORS

Open
#4,679 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
2.6k
Forks
1k
Avg merge
4d 15h
Merged PRs (30d)
19

Description

Describe your environment

OS: MacOS Sequoia
Python version: 3.11
SDK version: 1.33.1
API version: 1.33.1

What happened?

AWS Distro of Openelemetry Python while initializing sets an opinionated list of propagators via the OTEL_PROPAGATORS environment variable to baggage,xray,tracecontext but the actual propagators configured are still the default tracecontext,baggage.

Steps to Reproduce
  1. Create a python virtual environment and install ADOT Python pip install aws-opentelemetry-distro
  2. Create a python script main.py with the following content:
    from opentelemetry import propagate
    
    # print each propagator name
    for p in propagate.get_global_textmap()._propagators:
        print(p.__class__.__name__)
    
  3. Run the script using the following command
    OTEL_PYTHON_DISTRO=aws_distro \
    OTEL_PYTHON_CONFIGURATOR=aws_configurator \
    opentelemetry-instrument python3 main.py
    
  4. Among the logs (ignore the export failures) you will see the following propagators which are the default ones from OpenTelemetry SDK:
    TraceContextTextMapPropagator
    W3CBaggagePropagator
    
Expected Result

The printed propagators should be the following:

W3CBaggagePropagator
AwsXRayPropagator
TraceContextTextMapPropagator
Actual Result
TraceContextTextMapPropagator
W3CBaggagePropagator
Additional context

My RCA is that even before a distro is loaded and initialized, the propagate module is loaded and this module level code is executed which configures the propagators to tracecontext,baggage. By the time the distro sets the OTEL_PROPAGATORS env variable it's already too late.

In ADOT Python, we had to force a reload of the propagate module so that the new propagators are picked up and initialized. https://github.com/aws-observability/aws-otel-python-instrumentation/pull/421

Would you like to implement a fix?

None

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the issue with the ADOT installation and the opentelemetry-instrument command shown. Read opentelemetry-api/src/opentelemetry/propagate/init.py, then compare its initialization timing with the distro setup and the reload workaround in aws-observability/aws-otel-python-instrumentation#421. Done means OTEL_PROPAGATORS is honored and the reproduction prints baggage, AWS X-Ray, and trace context propagators.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.