grpc / grpc/grpc-java

grpc client sometimes stuck after connection reset

Offen
#3,427 6 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
bug
Vorherrschende Sprache
Java
Sterne
12.1k
Forks
4k
Ø Merge
2 T. 17 Std.
Gemergte PRs (30 T.)
37

Beschreibung

Please answer these questions before submitting your issue.

### What version of gRPC are you using?

1.2

### What JVM are you using (`java -version`)?

java version "1.8.0_121"
Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)

### What did you do?
If possible, provide a recipe for reproducing the error.

We run the services in kubernetes load balanced by a kubernetes. The servers are running grpc java server with TLS. Clients are also running grpc java. After we have changed a kubernetes setting of --conntrack-tcp-timeout-close-wait duration from default 1h0m0s to 60s and --conntrack-tcp-timeout-established duration from default 24h0m0s to 15m. We have started to see a lot of stuck clients where all calls timed out after seeing this exception:

```
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:288)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1100)
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:367)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:118)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:642)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:565)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:479)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:441)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
at java.lang.Thread.run(Thread.java:745)
```

### What did you expect to see?

All our calls are unary calls. I expect them they will go through even after seeing the connection reset.

### What did you see instead?

About 0.5% of clients are stuck (we have about 10k clients running). And inspecting the heap dump of a stuck client after the calls is made revealed the state 2 instances of InternalSubchannel and activeTransport and pendingTransport are null in them. Forcing the client call did not change the state of the InternalSubchannel, the client was stuck indefinitely until the process was killed and restarted. After restarting the process it became normal again.

Beitragsleitfaden

Beitragsleitfaden öffnen

Rechercherichtung

Beginne bei InternalSubchannel und dem im Stacktrace gezeigten Netty-Socket-Read-Pfad und reproduziere das Problem anschließend mit den beschriebenen Kubernetes-conntrack-Timeout-Einstellungen. Als erledigt gilt die Aufgabe, wenn unäre Aufrufe nach einem Verbindungsreset wiederhergestellt werden und Clients nicht länger auf unbestimmte Zeit mit null activeTransport und pendingTransport hängen bleiben.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
grpc, java
Bereich
backend-api-design, distributed-systems, networking
Issue-Typ
Bug
Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Aktivitätsstatus
Veraltet
Klarheit
Größtenteils klar
Anfängerfreundlichkeit
35/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.