grpc / grpc/grpc-java

Graceful shutdown within servlet container results in NoClassDefFoundError being thrown asynchronously

オープン
#9,510 コメント 12 件 リアクション 0 件 担当者 0 名 GitHub で見る
bug
主要言語
Java
スター
12.1k
フォーク
4k
平均マージ
2日 17時間
マージ済み PR(30日)
37

説明

### What version of gRPC-Java are you using?

`io.grpc:grpc-netty-shaded:1.46.0`

### What is your environment?

macOS / Linux / JDK 15 / Jetty 10

### What did you expect to see?

No exception being thrown after a graceful shutdown. Graceful shutdown happens during undeployment of web application.

### What did you see instead?

The log indicates that graceful shutdown has succeeded and consequently the web application has been undeployed. However, roughly a second later Netty throws `NoClassDefFoundException` seen below. Seems like `channel.awaitTermination()` returns too early.

```
06.09.2022 14:07:47.465 | grpc-shared-destroyer-0 | ERROR io.grpc.internal.LogExceptionRunnable
Exception while executing runnable io.grpc.internal.SharedResourceHolder$2@58725740
java.lang.ClassNotFoundException: io.grpc.netty.shaded.io.netty.util.concurrent.DefaultPromise$1
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:606)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:168)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
at org.eclipse.jetty.webapp.WebAppClassLoader.loadClass(WebAppClassLoader.java:511)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
... 17 common frames omitted
Wrapped by: java.lang.NoClassDefFoundError: io/grpc/netty/shaded/io/netty/util/concurrent/DefaultPromise$1
at io.grpc.netty.shaded.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:499)
at io.grpc.netty.shaded.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:616)
at io.grpc.netty.shaded.io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:609)
at io.grpc.netty.shaded.io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:117)
at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor.ensureThreadStarted(SingleThreadEventExecutor.java:961)
at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor.shutdownGracefully(SingleThreadEventExecutor.java:660)
at io.grpc.netty.shaded.io.netty.util.concurrent.MultithreadEventExecutorGroup.shutdownGracefully(MultithreadEventExecutorGroup.java:163)
at io.grpc.netty.shaded.io.grpc.netty.Utils$DefaultEventLoopGroupResource.close(Utils.java:439)
at io.grpc.netty.shaded.io.grpc.netty.Utils$DefaultEventLoopGroupResource.close(Utils.java:402)
at io.grpc.internal.SharedResourceHolder$2.run(SharedResourceHolder.java:139)
at io.grpc.internal.LogExceptionRunnable.run(LogExceptionRunnable.java:43)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
at java.base/java.lang.Thread.run(Thread.java:832)
```

### Steps to reproduce the bug

Use a servlet container, e.g. Jetty, and register a `ServletContextListener` that starts and stops a GRPC client to call an external service. When undeploying the web application, the exception is sometimes thrown.

```java
public final class MyGrpcServiceClient {

private ManagedChannel channel;

public void start() {
checkState(channel == null, "already started");
chanel = ManagedChannelBuilder.forTarget("dns:///.....").usePlaintext().build();
}

public void stop() {
checkState(channel != null, "already stopped")
channel.shutdown();
try {
if (!channel.awaitTermination(1, TimeUnit.MINUTES)) {
// log warning that graceful shutdown has taken too long
}
} catch (InterruptedException e) {
// log that graceful shutdown was interrupted
} finally {
channel = null;
}
}

// ... factory methods for returning new GRPC service stubs using above channel ...
}

public final class MyGrpcServiceClientSCL implements ServletContextListener {

private MyGrpcServiceClient client = new MyGrpcServiceClient();

@Override
public void contextInitialized(ServletContextEvent e) {
try {
client.start();
// log successful start of service client
} catch (Exception e) {
throw new Error(e);
}
}

@Override
public void contextDestroyed(ServletContextEvent e) {
try {
client.stop();
// log successful shutdown of service client
} catch (Exception e) {
// log unexpected exceptions
}
}

}

```

コントリビューションガイド

コントリビューションガイドを開く

調査の方向性

ServletContextListener/MyGrpcServiceClient の再現から始め、trace で示されているシャットダウンパス、特に SharedResourceHolder と Utils$DefaultEventLoopGroupResource を調査します。channel.awaitTermination と、servlet の undeployment 中に後から行われる非同期クリーンアップを比較します。遅延した NoClassDefFoundError や関連する例外が発生せずに正常なシャットダウンが完了すれば完了です。

索引モデルが issue の本文から書いたものです。

評価

技術スタック
java
領域
api, backend
issue の種類
バグ
難易度
4/5
見積もり時間
3〜5日
活発さ
停滞
明瞭さ
説明が足りない
初心者へのやさしさ
25/100

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。