open-telemetry / open-telemetry/opentelemetry-python-contrib
[grpc] AIO server spans record uncaught exceptions twice
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 1.1k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 16
Description
Describe your environment
- Python: 3.12.13
- grpcio: 1.78.1
- opentelemetry-api/sdk: 1.44.0
- opentelemetry-instrumentation-grpc: 0.65b0
- gRPC API:
grpc.aioserver
The current main implementation appears to have the same behavior.
What happened?
When a unary AIO gRPC handler raises an uncaught exception subclass (for example, ValueError or aiohttp.ClientConnectorError), the server span contains two identical exception events.
The exception is recorded once explicitly in _intercept_aio_server_unary:
except Exception as error:
if type(error) != Exception:
span.record_exception(error)
raise error
It is then recorded again when the exception leaves the context returned by _start_span(). _start_span() calls start_as_current_span() with set_status_on_exception=False, but leaves record_exception=True (the default).
The same pattern exists in _intercept_aio_server_stream.
Steps to reproduce
- Instrument a
grpc.aioserver withGrpcAioInstrumentorServer. - Configure an SDK tracer provider and an in-memory or OTLP exporter.
- Invoke a unary handler that raises an exception subclass.
- Inspect the exported server span events.
Observed result:
span.events = [exception, exception]
Both events contain the same exception type, message, and stack trace. One stack trace originates at the explicit span.record_exception(error) call; the other originates from opentelemetry.trace.use_span() when the re-raised exception exits start_as_current_span().
Expected behavior
A single uncaught exception should produce one exception event on the gRPC server span.
Suggested fix
Pass record_exception=False when creating the server span because the interceptor already records qualifying exceptions explicitly:
self._tracer.start_as_current_span(
...,
record_exception=False,
set_status_on_exception=False,
)
Alternatively, remove the explicit span.record_exception(error) and rely on the context manager, provided the existing special handling for bare Exception/gRPC aborts is preserved.
Additional context
Relevant source:
- AIO interceptor: https://github.com/open-telemetry/opentelemetry-python-contrib/blob/main/instrumentation/opentelemetry-instrumentation-grpc/src/opentelemetry/instrumentation/grpc/_aio_server.py
- Server span creation: https://github.com/open-telemetry/opentelemetry-python-contrib/blob/main/instrumentation/opentelemetry-instrumentation-grpc/src/opentelemetry/instrumentation/grpc/_server.py
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the AIO interceptor in instrumentation/opentelemetry-instrumentation-grpc/src/opentelemetry/instrumentation/grpc/_aio_server.py and server span creation in _server.py. Trace unary and stream exception handling, then reproduce an uncaught exception and verify the exported span contains one exception event while preserving the existing bare Exception and gRPC abort handling.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 82/100