grpc / grpc/grpc-java

grpc client sometimes stuck after connection reset

Open
#3,427 6 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
12.1k
Forks
4k
Avg merge
2d 17h
Merged PRs (30d)
37

Description

Please answer these questions before submitting your issue.

### What version of gRPC are you using?

1.2

### What JVM are you using (`java -version`)?

java version "1.8.0_121"
Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)

### What did you do?
If possible, provide a recipe for reproducing the error.

We run the services in kubernetes load balanced by a kubernetes. The servers are running grpc java server with TLS. Clients are also running grpc java. After we have changed a kubernetes setting of --conntrack-tcp-timeout-close-wait duration from default 1h0m0s to 60s and --conntrack-tcp-timeout-established duration from default 24h0m0s to 15m. We have started to see a lot of stuck clients where all calls timed out after seeing this exception:

```
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:288)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1100)
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:367)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:118)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:642)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:565)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:479)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:441)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
at java.lang.Thread.run(Thread.java:745)
```

### What did you expect to see?

All our calls are unary calls. I expect them they will go through even after seeing the connection reset.

### What did you see instead?

About 0.5% of clients are stuck (we have about 10k clients running). And inspecting the heap dump of a stuck client after the calls is made revealed the state 2 instances of InternalSubchannel and activeTransport and pendingTransport are null in them. Forcing the client call did not change the state of the InternalSubchannel, the client was stuck indefinitely until the process was killed and restarted. After restarting the process it became normal again.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.