Calls fail with Nginx, even with Transparent Retries
- Dominant language
- Java
- Stars
- 12.1k
- Forks
- 4k
- Avg merge
- 2d 17h
- Merged PRs (30d)
- 37
Description
I have a client that invokes a gRPC service through Nginx, and so I have to contend with the abrupt GOAWAYS sent by nginx every keepalive_requests.
My understanding is that transparent retry should handle this case. Originally I was on 1.54.1 and saw I was hitting: https://github.com/grpc/grpc-java/issues/10011 so I upgraded to 1.54.2. I'm still getting errors, however.
What confuses me a bit is that I see the original retry code was added here: https://github.com/grpc/grpc-java/commit/a83f67a706fdf9cd5b5c5026f1c413fbad1af839 and that there was a subsequent update for netty: https://github.com/grpc/grpc-java/commit/431fb0255fbed3f46ff3aca15ab37273c6143a47
I did some more tracing and now it seems like I'm hitting this path: https://github.com/grpc/grpc-java/blob/master/netty/src/main/java/io/grpc/netty/NettyClientHandler.java#L846
Why are we returning a code REFUSED instead of MISCARRIED? Or at a higher level, why doesn't a REFUSED stream get infinite transparent retries? https://github.com/grpc/grpc-java/blame/master/core/src/main/java/io/grpc/internal/RetriableStream.java#L937
And secondarily, how do we end up in the situation where we're getting GOAWAYs on sent streams? It seems like something to do with http2_max_concurrent_streams > nginx's keepalive_requests
Contributor guide
Assessment
This issue has not been assessed yet.