open-telemetry / open-telemetry/opentelemetry-python-contrib
Thread-ConsumeBidirectionalStream caught unexpected exception 'generator' object has no attribute 'add_done_callback' and will exit.
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 1.1k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 16
Description
Notes
We are trying to use Opentelemetry to instrument grpc to get traces of functions. We observed this error on local once, and we couldn't reproduce it again, but it's always on production server. Furthermore, we are using GCP monitoring suite (stack drive) to check traces.
Environment:
Django==3.2.6
Python==3.7
grpcio-tools==1.32.0
grpcio==1.33.2
opentelemetry-api==1.4.1
opentelemetry-sdk==1.4.1
opentelemetry-instrumentation-grpc==0.23b2
Instrumentation code:
set_global_textmap(CloudTraceFormatPropagator())
# sampling sends a portion of generated spans based on a 1/X population size
sample_rate = int(os.getenv('SAMPLE_RATE', '1'))
sampler = TraceIdRatioBased(1 / sample_rate)
tracer_provider = TracerProvider(sampler=sampler)
cloud_trace_exporter = CloudTraceSpanExporter()
tracer_provider.add_span_processor(SimpleSpanProcessor(cloud_trace_exporter))
trace.set_tracer_provider(tracer_provider)
tracer = trace.get_tracer(name)
GrpcInstrumentorClient().instrument() # when commented error isn't reproduced
except (ValueError, NotImplementedError, DefaultCredentialsError) as exc:
print(exc) # Handle errors here
The error
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/google/api_core/bidi.py", line 637, in _thread_main
self._bidi_rpc.open()
File "/usr/local/lib/python3.7/site-packages/google/api_core/bidi.py", line 287, in open
call._wrapped.add_done_callback(self._on_call_done)
AttributeError: 'generator' object has no attribute 'add_done_callback'
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the traceback in google/api_core/bidi.py, especially _thread_main and open, then inspect how GrpcInstrumentorClient().instrument() affects the bidirectional call. Compare the listed Python, grpcio, google-api-core, and OpenTelemetry versions and try to reproduce the production-only failure. Done means the interaction is explained and the reported exception no longer occurs in the relevant call path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- grpc, python
- Domain
- backend, observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100