grpc client sometimes stuck after connection reset

Please answer these questions before submitting your issue.

### What version of gRPC are you using?

1.2

### What JVM are you using (`java -version`)?

java version "1.8.0_121"
Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)

### What did you do?
If possible, provide a recipe for reproducing the error.

We run the services in kubernetes load balanced by a kubernetes. The servers are running grpc java server with TLS. Clients are also running grpc java. After we have changed a kubernetes setting of --conntrack-tcp-timeout-close-wait duration from default 1h0m0s to 60s and --conntrack-tcp-timeout-established duration from default 24h0m0s to 15m. We have started to see a lot of stuck clients where all calls timed out after seeing this exception:

```
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:288)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1100)
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:367)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:118)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:642)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:565)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:479)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:441)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
at java.lang.Thread.run(Thread.java:745)
```

### What did you expect to see?

All our calls are unary calls. I expect them they will go through even after seeing the connection reset.

### What did you see instead?

About 0.5% of clients are stuck (we have about 10k clients running). And inspecting the heap dump of a stuck client after the calls is made revealed the state 2 instances of InternalSubchannel and activeTransport and pendingTransport are null in them. Forcing the client call did not change the state of the InternalSubchannel, the client was stuck indefinitely until the process was killed and restarted. After restarting the process it became normal again.

Start at InternalSubchannel and the Netty socket-read path shown in the stack trace, then reproduce with the Kubernetes conntrack timeout settings described. Done means unary calls recover after a connection reset and clients no longer remain indefinitely stuck with null activeTransport and pendingTransport.

Scritto dal modello di indicizzazione a partire dal testo della issue.