grpc / grpc/grpc-java

Graceful shutdown within servlet container results in NoClassDefFoundError being thrown asynchronously

Open
#9,510 12 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
12.1k
Forks
4k
Avg merge
2d 17h
Merged PRs (30d)
37

Description

### What version of gRPC-Java are you using?

`io.grpc:grpc-netty-shaded:1.46.0`

### What is your environment?

macOS / Linux / JDK 15 / Jetty 10

### What did you expect to see?

No exception being thrown after a graceful shutdown. Graceful shutdown happens during undeployment of web application.

### What did you see instead?

The log indicates that graceful shutdown has succeeded and consequently the web application has been undeployed. However, roughly a second later Netty throws `NoClassDefFoundException` seen below. Seems like `channel.awaitTermination()` returns too early.

```
06.09.2022 14:07:47.465 | grpc-shared-destroyer-0 | ERROR io.grpc.internal.LogExceptionRunnable
Exception while executing runnable io.grpc.internal.SharedResourceHolder$2@58725740
java.lang.ClassNotFoundException: io.grpc.netty.shaded.io.netty.util.concurrent.DefaultPromise$1
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:606)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:168)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
at org.eclipse.jetty.webapp.WebAppClassLoader.loadClass(WebAppClassLoader.java:511)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
... 17 common frames omitted
Wrapped by: java.lang.NoClassDefFoundError: io/grpc/netty/shaded/io/netty/util/concurrent/DefaultPromise$1
at io.grpc.netty.shaded.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:499)
at io.grpc.netty.shaded.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:616)
at io.grpc.netty.shaded.io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:609)
at io.grpc.netty.shaded.io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:117)
at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor.ensureThreadStarted(SingleThreadEventExecutor.java:961)
at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor.shutdownGracefully(SingleThreadEventExecutor.java:660)
at io.grpc.netty.shaded.io.netty.util.concurrent.MultithreadEventExecutorGroup.shutdownGracefully(MultithreadEventExecutorGroup.java:163)
at io.grpc.netty.shaded.io.grpc.netty.Utils$DefaultEventLoopGroupResource.close(Utils.java:439)
at io.grpc.netty.shaded.io.grpc.netty.Utils$DefaultEventLoopGroupResource.close(Utils.java:402)
at io.grpc.internal.SharedResourceHolder$2.run(SharedResourceHolder.java:139)
at io.grpc.internal.LogExceptionRunnable.run(LogExceptionRunnable.java:43)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
at java.base/java.lang.Thread.run(Thread.java:832)
```

### Steps to reproduce the bug

Use a servlet container, e.g. Jetty, and register a `ServletContextListener` that starts and stops a GRPC client to call an external service. When undeploying the web application, the exception is sometimes thrown.

```java
public final class MyGrpcServiceClient {

private ManagedChannel channel;

public void start() {
checkState(channel == null, "already started");
chanel = ManagedChannelBuilder.forTarget("dns:///.....").usePlaintext().build();
}

public void stop() {
checkState(channel != null, "already stopped")
channel.shutdown();
try {
if (!channel.awaitTermination(1, TimeUnit.MINUTES)) {
// log warning that graceful shutdown has taken too long
}
} catch (InterruptedException e) {
// log that graceful shutdown was interrupted
} finally {
channel = null;
}
}

// ... factory methods for returning new GRPC service stubs using above channel ...
}

public final class MyGrpcServiceClientSCL implements ServletContextListener {

private MyGrpcServiceClient client = new MyGrpcServiceClient();

@Override
public void contextInitialized(ServletContextEvent e) {
try {
client.start();
// log successful start of service client
} catch (Exception e) {
throw new Error(e);
}
}

@Override
public void contextDestroyed(ServletContextEvent e) {
try {
client.stop();
// log successful shutdown of service client
} catch (Exception e) {
// log unexpected exceptions
}
}

}

```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.