grpc / grpc/grpc-java

Retries on xds:/// channels (or with RawMessageClientInterceptor) send empty 0-byte payload on retry

Open
#13,010 2 comments 0 reactions 1 assignee Claimed by @kannanjgithub View on GitHub
Dominant language
Java
Stars
12.1k
Forks
4k
Avg merge
2d 17h
Merged PRs (30d)
37

Description

### What version of gRPC-Java are you using?
Broken: 1.83.0/1.83.1
Working: 1.82.2

### What is your environment?
Azul Zulu JDK 26

### Summary
When an RPC is retried on an 'xds:///' channel (or any channel with raw 'InputStream' pass-through marshalling), retry attempts send an empty 0-byte payload (a 5-byte gRPC frame '00 00 00 00 00') rather than replying with the original request payload. If the backend server is configured to also accept an empty protobuf message (e.g. a batch request containing 0 items) and returns 'OK', this results in silent false positives, ie the replayed (now) empty message doesn't throw any error.

RetriableStream replays messages via 'method.streamRequest(message)'. When 'method' uses a raw 'InputStream' pass-through marshaller, the already consumed 'InputStream' is passed to subsequent retry attempts without being re-buffered. Starting on version 1.83.0, 'XdsNameResolver' introduced 'RawMessageClientInterceptor' on every 'xds:///' call. In turn, that rebuilds the method with a pass-through InputStream marshaller. As a result, all retried calls on 'xds:///' channels transmit empty request bodies.
The retry itself comes from Traffic Director over xDS.

### What did you expect to see?
On retry, the client should replay the full original payload buffer
user-agent: grpc-java-netty/1.83.0
Attempt 1 (StreamId=3): sends 857 bytes -> server returns grpc-status: 13
Retry 1 (streamId=3 on second connection): sends 857 bytes -> server returns grpc-status: 13
Retry 2 (streamId=5): sends 857 bytes -> server returns grpc-status: 13
Retry 3 (streamId=5): sends 857 bytes -> server returns grpc-status: 13
Retry 4 (streamId=7): sends 857 bytes -> server returns grpc-status: 13
StatusRuntimeException(INTERNAL) -> MyServiceInternalException

### What did you see instead?
user-agent: grpc-java-netty/1.83.0
Attempt 1 (StreamId=3): sends 857 bytes -> server returns grpc-status: 13
Retry (streamId=5): sends 5 bytes (0000000000 - empty body) -> server returns grpc-status: 0 (OK) with empty response

### Steps to reproduce the bug
1. Using grpc-java 1.83.0+, create an xds:// channel (or a channel with XdsNameResolver.RawMessageClientInterceptor) with RetryPolicy configured in serviceConfig
2. Issue a unary or batch RPC (with a non-empty payload)
3. Have the server return a retryable status (INTERNAL/UNAVAILABLE) on attempt 1
4. Attempt 2 will transmit an empty 5-byte frame with 0 payload bytes.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.