temporalio / temporalio/sdk-python

[Bug] Unable to run workflows with OpenTelemetry and ddtrace

Open
#733 10 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
1.2k
Forks
241
Avg merge
3d 21h
Merged PRs (30d)
55

Description

What are you really trying to do?

I'm attempting to enable tracing on my worker using the TracingInterceptor and the tracer provided by ddtrace.

Describe the bug

A change upstream with the OpenTelemetry SDK seems to have caused an issue with Temporal's sandbox environment, causing an error stating os.environ.get isn't allowed in a workflow.

The relevant change on the OpenTelemetry side is here - if I run with a version prior to 1.29 then I no longer get an error.

Minimal Reproduction

A workflow implemented on a client with the TracingInterceptor attached and a valid OpenTelemetry SDK configured, in my case I'm using ddtrace with the DD_TRACE_OTEL_ENABLED environment variable set to true. From what I can tell though the call that isn't allowed in the sandbox is coming directly from the OpenTelemetry SDK and not the ddtrace patch.

Environment/Versions
  • OS and processor: macOS Apple Silicon
  • Temporal Version: SDK 1.9.0
  • Running in Docker arm64
Additional context

You can see the stacktrace and error that's displayed on the Temporal UI here: https://gist.github.com/connected-bkiiskila/d11592cb86271b5f343a4d387097d418

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the TracingInterceptor and the Temporal sandbox using the minimal workflow reproduction described in the issue. Run it with the OpenTelemetry SDK at version 1.29 or later and compare it with a version before 1.29, then inspect the linked stack trace and upstream OpenTelemetry change. Done means the traced workflow runs without the sandbox rejecting os.environ.get.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.