open-telemetry / open-telemetry/opentelemetry-python-contrib

"Failed to detach context" exception for grpc instrumentation

Open
#2,357 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
1.1k
Forks
1.1k
Avg merge
4d 15h
Merged PRs (30d)
16

Description

open telemetry version - 0.44b0
python - 3.10
docker versions - ubuntu 20.04
running in Kubernetes cluster, on prem servers

The issue isn't reproduced systematically
many times i get the following exception (everything is fine except for the error printed)

Failed to detach context
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/opentelemetry/trace/__init__.py", line 573, in use_span
    yield span
  File "/usr/local/lib/python3.10/dist-packages/opentelemetry/sdk/trace/__init__.py", line 1046, in start_as_current_span
    yield span
  File "/usr/local/lib/python3.10/dist-packages/opentelemetry/instrumentation/grpc/_client.py", line 211, in _intercept_server_stream
    yield from invoker(request_or_iterator, metadata)
GeneratorExit

During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/opentelemetry/context/__init__.py", line 163, in detach
    _RUNTIME_CONTEXT.detach(token)  # type: ignore
  File "/usr/local/lib/python3.10/dist-packages/opentelemetry/context/contextvars_context.py", line 50, in detach
    self._current_context.reset(token)  # type: ignore
ValueError: <Token var=<ContextVar name='current_context' default={} at 0x7fbf8d8b2bb0> at 0x7fbf4c4b9d40> was created in a different Context

there are related bugs but not regarding grpc instrumentation
https://github.com/open-telemetry/opentelemetry-python-contrib/issues/904
https://github.com/open-telemetry/opentelemetry-python/issues/2606

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading opentelemetry/instrumentation/grpc/_client.py, especially _intercept_server_stream, alongside the traceback from context/init.py and contextvars_context.py. Investigate the reported Python 3.10, Kubernetes, and server-streaming environment and compare the related issues. Done means the intermittent “Failed to detach context” exception is understood and no longer printed during otherwise successful grpc instrumentation.

Written by the indexing model from the issue text.

Assessment

Tech stack
grpc, python
Domain
api, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.