open-telemetry / open-telemetry/opentelemetry-python-contrib

aws orig error in aks environiment in otel auto instrumentation python setup

Open
#3,182 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
1.1k
Forks
1.1k
Avg merge
4d 15h
Merged PRs (30d)
16

Description

What happened?

I'm setting up distributed tracing and end-to-end transaction monitoring for my services, which are hosted in an AKS (Azure Kubernetes Service) cluster and built using the Python framework. Since we use a microservices architecture, I'm implementing OpenTelemetry (OTel) with auto-instrumentation. This involves deploying the OTel Operator in AKS along with an OTel Collector and using Jaeger for tracing and ELK as backend storage

As part of the setup, I added the necessary Python-related annotations to enable tracing for four of my services. However, while I can see trace data for two of the services, the other two are not sending any data. Instead, I see an error message:

Could not find the ORIG_HANDLER or _HANDLER in the environment variables. This instrumentation requires the OpenTelemetry Lambda extension installed.

Since our environment is in Azure and not AWS, I don't understand why this error is appearing. What could be causing this issue, and how can I resolve it?

Steps to Reproduce

setup open telemetry by deploying otel operator in aks with this command

helm install opentelemetry-operator open-telemetry/opentelemetry-operator --namespace opentelemetry-operator --create-namespace --set "manager.collectorImage.repository=otel/opentelemetry-collector-k8s"
--set admissionWebhooks.certManager.enabled=false
--set admissionWebhooks.autoGenerateCert.enabled=true

and deploy otel collector and instrumentation.yaml wity the following

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
name: traces
spec:
mode: deployment
config:
receivers:
otlp:
protocols:
grpc: {}
http: {}
processors:
memory_limiter:
check_interval: 1s
limit_percentage: 75
spike_limit_percentage: 15
batch:
send_batch_size: 10000
timeout: 10s
exporters:
otlp:

endpoint: "jaeger-collector:4317"

endpoint: "jaeger-collector.observability.svc.cluster.local:4317"

endpoint: "jaeger-collector.jaeger.svc.cluster.local:4317"

tls:
insecure: true
debug: {}
service:
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [debug, otlp]

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
name: otel-instrumentation
spec:
exporter:
endpoint: http://traces-collector.opentelemetry-operator.svc.cluster.local:4317/
propagators:

  • baggage
  • b3
    sampler:
    type: parentbased_traceidratio
    argument: "1"
    python:
    env:
  • name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: http://traces-collector.opentelemetry-operator.svc.cluster.local:4318/v1/traces
  • name: OTEL_PYTHON_DISABLED_INSTRUMENTATIONS
    value: aws-lambda,metrics
  • name: OTEL_METRICS_EXPORTER
    value: none
  • name: OTEL_LOG_LEVEL
    value: "debug"
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:latest
Expected Result

need to get traces exported from the services of aks to the jaeger ui backend but not able to get that

Actual Result

i'm getting this error

Could not find the ORIG_HANDLER or _HANDLER in the environment variables. This instrumentation requires the OpenTelemetry Lambda extension installed.

this is aws labda error which is not related to aks or azure but i'm getting this error in the logs of services which i auto instrumenetd with these annotations

instrumentation.opentelemetry.io/inject-python:true
instrumentation.opentelemetry.io/otel-python-platform:musl
instrumentation.opentelemetry.io/container-names

Additional context

No response

Would you like to implement a fix?

None

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by comparing the auto-instrumentation annotations, environment variables, and logs of the four AKS services, focusing on the two that do not emit traces. Review the Python instrumentation configuration and the reported Lambda-related error, then verify completion by confirming that traces from all services reach the configured collector and Jaeger backend.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, helm, kubernetes, python
Domain
backend, cloud, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.