googleapis / googleapis/google-cloud-rust
Capture stream lifetime errors in tracing spans
- Dominant language
- Rust
- Stars
- 955
- Forks
- 144
- Avg merge
- 1d 5h
- Merged PRs (30d)
- 279
Description
The current implementation of observability for streaming RPCs (specifically in `bidi_stream_with_status` and `server_streaming_with_status` in `gax-internal/src/grpc.rs`) only instruments the future that establishes the stream using `WithTransportSpan`.
This means:
1. The span only measures the duration of the *initiation* of the stream.
2. The span is closed with `OK` status as soon as the stream is successfully opened.
3. Any errors that occur later during the streaming phase (e.g., when reading/writing messages) are not captured on the span.
This creates a problem: any errors that occur after the stream is successfully opened (during data transmission) are not recorded on the span. The span remains marked as successful even if the stream later fails.
This was observed in the `grpc_observability.rs` tests (`streaming_error`), where a stream error caused by an invalid message did not result in the span being marked as an error because the connection itself already succeeded.
We should instrument the stream itself (not just the initial connection future) so that errors encountered during the stream's lifetime are recorded on the span. This might involve wrapping the returned stream in a custom type that can intercept errors and update the span status accordingly.
Contributor guide
Assessment
This issue has not been assessed yet.