Retries on xds:/// channels (or with RawMessageClientInterceptor) send empty 0-byte payload on retry
- Dominant language
- Java
- Stars
- 12.1k
- Forks
- 4k
- Avg merge
- 2d 17h
- Merged PRs (30d)
- 37
Description
### What version of gRPC-Java are you using?
Broken: 1.83.0/1.83.1
Working: 1.82.2
### What is your environment?
Azul Zulu JDK 26
### Summary
When an RPC is retried on an 'xds:///' channel (or any channel with raw 'InputStream' pass-through marshalling), retry attempts send an empty 0-byte payload (a 5-byte gRPC frame '00 00 00 00 00') rather than replying with the original request payload. If the backend server is configured to also accept an empty protobuf message (e.g. a batch request containing 0 items) and returns 'OK', this results in silent false positives, ie the replayed (now) empty message doesn't throw any error.
RetriableStream replays messages via 'method.streamRequest(message)'. When 'method' uses a raw 'InputStream' pass-through marshaller, the already consumed 'InputStream' is passed to subsequent retry attempts without being re-buffered. Starting on version 1.83.0, 'XdsNameResolver' introduced 'RawMessageClientInterceptor' on every 'xds:///' call. In turn, that rebuilds the method with a pass-through InputStream marshaller. As a result, all retried calls on 'xds:///' channels transmit empty request bodies.
The retry itself comes from Traffic Director over xDS.
### What did you expect to see?
On retry, the client should replay the full original payload buffer
user-agent: grpc-java-netty/1.83.0
Attempt 1 (StreamId=3): sends 857 bytes -> server returns grpc-status: 13
Retry 1 (streamId=3 on second connection): sends 857 bytes -> server returns grpc-status: 13
Retry 2 (streamId=5): sends 857 bytes -> server returns grpc-status: 13
Retry 3 (streamId=5): sends 857 bytes -> server returns grpc-status: 13
Retry 4 (streamId=7): sends 857 bytes -> server returns grpc-status: 13
StatusRuntimeException(INTERNAL) -> MyServiceInternalException
### What did you see instead?
user-agent: grpc-java-netty/1.83.0
Attempt 1 (StreamId=3): sends 857 bytes -> server returns grpc-status: 13
Retry (streamId=5): sends 5 bytes (0000000000 - empty body) -> server returns grpc-status: 0 (OK) with empty response
### Steps to reproduce the bug
1. Using grpc-java 1.83.0+, create an xds:// channel (or a channel with XdsNameResolver.RawMessageClientInterceptor) with RetryPolicy configured in serviceConfig
2. Issue a unary or batch RPC (with a non-empty payload)
3. Have the server return a retryable status (INTERNAL/UNAVAILABLE) on attempt 1
4. Attempt 2 will transmit an empty 5-byte frame with 0 payload bytes.
Contributor guide
Assessment
This issue has not been assessed yet.