grpc / grpc/grpc-java

Calls fail with Nginx, even with Transparent Retries

Open
#10,533 11 comments 0 reactions 1 assignee Claimed by @larry-safran View on GitHub
question
Dominant language
Java
Stars
12.1k
Forks
4k
Avg merge
2d 17h
Merged PRs (30d)
37

Description

I have a client that invokes a gRPC service through Nginx, and so I have to contend with the abrupt GOAWAYS sent by nginx every keepalive_requests.

My understanding is that transparent retry should handle this case. Originally I was on 1.54.1 and saw I was hitting: https://github.com/grpc/grpc-java/issues/10011 so I upgraded to 1.54.2. I'm still getting errors, however.

What confuses me a bit is that I see the original retry code was added here: https://github.com/grpc/grpc-java/commit/a83f67a706fdf9cd5b5c5026f1c413fbad1af839 and that there was a subsequent update for netty: https://github.com/grpc/grpc-java/commit/431fb0255fbed3f46ff3aca15ab37273c6143a47

I did some more tracing and now it seems like I'm hitting this path: https://github.com/grpc/grpc-java/blob/master/netty/src/main/java/io/grpc/netty/NettyClientHandler.java#L846

Why are we returning a code REFUSED instead of MISCARRIED? Or at a higher level, why doesn't a REFUSED stream get infinite transparent retries? https://github.com/grpc/grpc-java/blame/master/core/src/main/java/io/grpc/internal/RetriableStream.java#L937

And secondarily, how do we end up in the situation where we're getting GOAWAYs on sent streams? It seems like something to do with http2_max_concurrent_streams > nginx's keepalive_requests

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.