open-telemetry / open-telemetry/opentelemetry-python-contrib

Thread-ConsumeBidirectionalStream caught unexpected exception 'generator' object has no attribute 'add_done_callback' and will exit.

Open
#1,180 9 comments 10 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
1.1k
Forks
1.1k
Avg merge
4d 15h
Merged PRs (30d)
16

Description

Notes
We are trying to use Opentelemetry to instrument grpc to get traces of functions. We observed this error on local once, and we couldn't reproduce it again, but it's always on production server. Furthermore, we are using GCP monitoring suite (stack drive) to check traces.

Environment:
Django==3.2.6
Python==3.7

grpcio-tools==1.32.0
grpcio==1.33.2

opentelemetry-api==1.4.1
opentelemetry-sdk==1.4.1
opentelemetry-instrumentation-grpc==0.23b2

Instrumentation code:
set_global_textmap(CloudTraceFormatPropagator())
# sampling sends a portion of generated spans based on a 1/X population size
sample_rate = int(os.getenv('SAMPLE_RATE', '1'))
sampler = TraceIdRatioBased(1 / sample_rate)
tracer_provider = TracerProvider(sampler=sampler)
cloud_trace_exporter = CloudTraceSpanExporter()
tracer_provider.add_span_processor(SimpleSpanProcessor(cloud_trace_exporter))
trace.set_tracer_provider(tracer_provider)
tracer = trace.get_tracer(name)
GrpcInstrumentorClient().instrument() # when commented error isn't reproduced
except (ValueError, NotImplementedError, DefaultCredentialsError) as exc:
print(exc) # Handle errors here

The error
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/google/api_core/bidi.py", line 637, in _thread_main
self._bidi_rpc.open()
File "/usr/local/lib/python3.7/site-packages/google/api_core/bidi.py", line 287, in open
call._wrapped.add_done_callback(self._on_call_done)
AttributeError: 'generator' object has no attribute 'add_done_callback'

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the traceback in google/api_core/bidi.py, especially _thread_main and open, then inspect how GrpcInstrumentorClient().instrument() affects the bidirectional call. Compare the listed Python, grpcio, google-api-core, and OpenTelemetry versions and try to reproduce the production-only failure. Done means the interaction is explained and the reported exception no longer occurs in the relevant call path.

Written by the indexing model from the issue text.

Assessment

Tech stack
grpc, python
Domain
backend, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.