baidu / baidu/Jprotobuf-rpc-socket
RpcClient#shutdown 极端情况可能会出现hang死情况
- Dominant language
- Java
- Stars
- 535
- Forks
- 216
- PR merge metrics
- No merged PRs in 30d
Description
**RpcClient**建立连接过程中是同步等待在这里:
```
at io.netty.util.concurrent.DefaultPromise.awaitUninterruptibly(DefaultPromise.java:286)
- locked <0x00000006506433d0> (a io.netty.channel.DefaultChannelPromise)
at io.netty.channel.DefaultChannelPromise.awaitUninterruptibly(DefaultChannelPromise.java:135)
at io.netty.channel.DefaultChannelPromise.awaitUninterruptibly(DefaultChannelPromise.java:28)
at com.baidu.jprotobuf.pbrpc.transport.ChannelPoolObjectFactory.wrap(ChannelPoolObjectFactory.java:75)
at com.baidu.jprotobuf.pbrpc.transport.ChannelPoolObjectFactory.wrap(ChannelPoolObjectFactory.java:35)
at org.apache.commons.pool2.BasePooledObjectFactory.makeObject(BasePooledObjectFactory.java:60)
at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:868)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
at com.baidu.jprotobuf.pbrpc.transport.ChannelPool.getChannel(ChannelPool.java:61)
at com.baidu.jprotobuf.pbrpc.transport.RpcChannel.getConnection(RpcChannel.java:64)
at com.baidu.jprotobuf.pbrpc.client.ProtobufRpcProxy.invoke(ProtobufRpcProxy.java:366)
```
如果上述连接未建立完成/连接失败,并且**RpcClient**过早暴露并被其他线程执行**shutdown**,上面的线程无法再被唤醒。
无法被唤醒的具体原因是依赖的**Netty 4.0.27.Final** shutdown 的时候没有处理未完成连接的channel等待的**DefaultChannelPromise**。最新版的Netty已经修复这个bug。
### Bug复现:
#### 环境信息
* Jprotobuf-rpc-socket:4.1.0
* 操作系统版本: Debian 4.19.98-1
* JDK版本:Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11.0.5+10-LTS, mixed mode)
```
import com.baidu.jprotobuf.pbrpc.EchoService;
import com.baidu.jprotobuf.pbrpc.client.ProtobufRpcProxy;
import com.baidu.jprotobuf.pbrpc.transport.RpcClient;
import org.junit.Test;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
/**
* Coded by senninha on 2020/5/12
*/
public class TestRpcHang {
@Test
public void testRpcHang() throws InterruptedException {
while (true) {
ExecutorService executorService = Executors.newSingleThreadExecutor();
final RpcClient rpcClient = new RpcClient();
// 创建EchoService代理
ProtobufRpcProxy pbrpcProxy = new ProtobufRpcProxy(rpcClient, EchoService.class);
pbrpcProxy.setPort(1080);
pbrpcProxy.setLookupStubOnStartup(true);
executorService.submit(new Runnable() {
@Override
public void run() {
rpcClient.shutdown();
}
});
try {
// 动态生成代理实例
EchoService echoService = pbrpcProxy.proxy();
} catch (Exception e) {
// ignore
}
System.err.println("Not hang!");
Thread.sleep(1000);
}
}
}
```
纯Netty版复现:
```
@Test
public void testNetty() {
ExecutorService executorService = Executors.newSingleThreadExecutor();
while (true) {
final Bootstrap bootstrap = new Bootstrap();
final NioEventLoopGroup group = new NioEventLoopGroup(1);
bootstrap.group(group);
bootstrap.channel(NioSocketChannel.class);
bootstrap.option(ChannelOption.SO_KEEPALIVE, true);
bootstrap.option(ChannelOption.CONNECT_TIMEOUT_MILLIS, 1000);
bootstrap.handler(new ChannelInitializer() {
@Override
public void initChannel(SocketChannel ch) {
ch.pipeline().addLast(new TimeClientHandler());
}
});
ChannelFuture connect = bootstrap.connect("localhost", 1080);
executorService.submit(new Runnable() {
@Override
public void run() {
group.shutdownGracefully();
}
});
try {
connect.awaitUninterruptibly();
} catch (Exception e) {
e.printStackTrace();
}
}
}
private static class TimeClientHandler extends ChannelInboundHandlerAdapter {
@Override
public void channelRead(ChannelHandlerContext ctx, Object msg) {
ByteBuf m = (ByteBuf) msg; // (1)
try {
long currentTimeMillis = (m.readUnsignedInt() - 2208988800L) * 1000L;
System.out.println(new Date(currentTimeMillis));
ctx.close();
} finally {
m.release();
}
}
@Override
public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) {
cause.printStackTrace();
ctx.close();
}
}
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with ChannelPoolObjectFactory.wrap, ChannelPool.getChannel, RpcChannel.getConnection, and RpcClient.shutdown from the stack trace. Run the TestRpcHang or testNetty reproductions under the stated Java and Jprotobuf-rpc-socket versions. Done means shutdown during an incomplete connection no longer leaves the waiting thread blocked indefinitely.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100