temporalio / temporalio/sdk-python
[Bug] Unable to run workflows with OpenTelemetry and ddtrace
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 241
- Avg merge
- 3d 21h
- Merged PRs (30d)
- 55
Description
What are you really trying to do?
I'm attempting to enable tracing on my worker using the TracingInterceptor and the tracer provided by ddtrace.
Describe the bug
A change upstream with the OpenTelemetry SDK seems to have caused an issue with Temporal's sandbox environment, causing an error stating os.environ.get isn't allowed in a workflow.
The relevant change on the OpenTelemetry side is here - if I run with a version prior to 1.29 then I no longer get an error.
Minimal Reproduction
A workflow implemented on a client with the TracingInterceptor attached and a valid OpenTelemetry SDK configured, in my case I'm using ddtrace with the DD_TRACE_OTEL_ENABLED environment variable set to true. From what I can tell though the call that isn't allowed in the sandbox is coming directly from the OpenTelemetry SDK and not the ddtrace patch.
Environment/Versions
- OS and processor: macOS Apple Silicon
- Temporal Version: SDK 1.9.0
- Running in Docker arm64
Additional context
You can see the stacktrace and error that's displayed on the Temporal UI here: https://gist.github.com/connected-bkiiskila/d11592cb86271b5f343a4d387097d418
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the TracingInterceptor and the Temporal sandbox using the minimal workflow reproduction described in the issue. Run it with the OpenTelemetry SDK at version 1.29 or later and compare it with a version before 1.29, then inspect the linked stack trace and upstream OpenTelemetry change. Done means the traced workflow runs without the sandbox rejecting os.environ.get.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100