open-telemetry / open-telemetry/opentelemetry-python-contrib

[grpc] AIO server spans record uncaught exceptions twice

Open Beginner friendly
#5,068 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.1k
Forks
1.1k
Avg merge
4d 15h
Merged PRs (30d)
16

Description

Describe your environment
  • Python: 3.12.13
  • grpcio: 1.78.1
  • opentelemetry-api/sdk: 1.44.0
  • opentelemetry-instrumentation-grpc: 0.65b0
  • gRPC API: grpc.aio server

The current main implementation appears to have the same behavior.

What happened?

When a unary AIO gRPC handler raises an uncaught exception subclass (for example, ValueError or aiohttp.ClientConnectorError), the server span contains two identical exception events.

The exception is recorded once explicitly in _intercept_aio_server_unary:

except Exception as error:
    if type(error) != Exception:
        span.record_exception(error)
    raise error

It is then recorded again when the exception leaves the context returned by _start_span(). _start_span() calls start_as_current_span() with set_status_on_exception=False, but leaves record_exception=True (the default).

The same pattern exists in _intercept_aio_server_stream.

Steps to reproduce
  1. Instrument a grpc.aio server with GrpcAioInstrumentorServer.
  2. Configure an SDK tracer provider and an in-memory or OTLP exporter.
  3. Invoke a unary handler that raises an exception subclass.
  4. Inspect the exported server span events.

Observed result:

span.events = [exception, exception]

Both events contain the same exception type, message, and stack trace. One stack trace originates at the explicit span.record_exception(error) call; the other originates from opentelemetry.trace.use_span() when the re-raised exception exits start_as_current_span().

Expected behavior

A single uncaught exception should produce one exception event on the gRPC server span.

Suggested fix

Pass record_exception=False when creating the server span because the interceptor already records qualifying exceptions explicitly:

self._tracer.start_as_current_span(
    ...,
    record_exception=False,
    set_status_on_exception=False,
)

Alternatively, remove the explicit span.record_exception(error) and rely on the context manager, provided the existing special handling for bare Exception/gRPC aborts is preserved.

Additional context

Relevant source:

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the AIO interceptor in instrumentation/opentelemetry-instrumentation-grpc/src/opentelemetry/instrumentation/grpc/_aio_server.py and server span creation in _server.py. Trace unary and stream exception handling, then reproduce an uncaught exception and verify the exported span contains one exception event while preserving the existing bare Exception and gRPC abort handling.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
observability
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
82/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.