Graceful shutdown within servlet container results in NoClassDefFoundError being thrown asynchronously
- Dominant language
- Java
- Stars
- 12.1k
- Forks
- 4k
- Avg merge
- 2d 17h
- Merged PRs (30d)
- 37
Description
### What version of gRPC-Java are you using?
`io.grpc:grpc-netty-shaded:1.46.0`
### What is your environment?
macOS / Linux / JDK 15 / Jetty 10
### What did you expect to see?
No exception being thrown after a graceful shutdown. Graceful shutdown happens during undeployment of web application.
### What did you see instead?
The log indicates that graceful shutdown has succeeded and consequently the web application has been undeployed. However, roughly a second later Netty throws `NoClassDefFoundException` seen below. Seems like `channel.awaitTermination()` returns too early.
```
06.09.2022 14:07:47.465 | grpc-shared-destroyer-0 | ERROR io.grpc.internal.LogExceptionRunnable
Exception while executing runnable io.grpc.internal.SharedResourceHolder$2@58725740
java.lang.ClassNotFoundException: io.grpc.netty.shaded.io.netty.util.concurrent.DefaultPromise$1
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:606)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:168)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
at org.eclipse.jetty.webapp.WebAppClassLoader.loadClass(WebAppClassLoader.java:511)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
... 17 common frames omitted
Wrapped by: java.lang.NoClassDefFoundError: io/grpc/netty/shaded/io/netty/util/concurrent/DefaultPromise$1
at io.grpc.netty.shaded.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:499)
at io.grpc.netty.shaded.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:616)
at io.grpc.netty.shaded.io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:609)
at io.grpc.netty.shaded.io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:117)
at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor.ensureThreadStarted(SingleThreadEventExecutor.java:961)
at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor.shutdownGracefully(SingleThreadEventExecutor.java:660)
at io.grpc.netty.shaded.io.netty.util.concurrent.MultithreadEventExecutorGroup.shutdownGracefully(MultithreadEventExecutorGroup.java:163)
at io.grpc.netty.shaded.io.grpc.netty.Utils$DefaultEventLoopGroupResource.close(Utils.java:439)
at io.grpc.netty.shaded.io.grpc.netty.Utils$DefaultEventLoopGroupResource.close(Utils.java:402)
at io.grpc.internal.SharedResourceHolder$2.run(SharedResourceHolder.java:139)
at io.grpc.internal.LogExceptionRunnable.run(LogExceptionRunnable.java:43)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
at java.base/java.lang.Thread.run(Thread.java:832)
```
### Steps to reproduce the bug
Use a servlet container, e.g. Jetty, and register a `ServletContextListener` that starts and stops a GRPC client to call an external service. When undeploying the web application, the exception is sometimes thrown.
```java
public final class MyGrpcServiceClient {
private ManagedChannel channel;
public void start() {
checkState(channel == null, "already started");
chanel = ManagedChannelBuilder.forTarget("dns:///.....").usePlaintext().build();
}
public void stop() {
checkState(channel != null, "already stopped")
channel.shutdown();
try {
if (!channel.awaitTermination(1, TimeUnit.MINUTES)) {
// log warning that graceful shutdown has taken too long
}
} catch (InterruptedException e) {
// log that graceful shutdown was interrupted
} finally {
channel = null;
}
}
// ... factory methods for returning new GRPC service stubs using above channel ...
}
public final class MyGrpcServiceClientSCL implements ServletContextListener {
private MyGrpcServiceClient client = new MyGrpcServiceClient();
@Override
public void contextInitialized(ServletContextEvent e) {
try {
client.start();
// log successful start of service client
} catch (Exception e) {
throw new Error(e);
}
}
@Override
public void contextDestroyed(ServletContextEvent e) {
try {
client.stop();
// log successful shutdown of service client
} catch (Exception e) {
// log unexpected exceptions
}
}
}
```
Contributor guide
Assessment
This issue has not been assessed yet.