apache / apache/dubbo

[Bug] 3.x Graceful Shutdown. "UNIMPLEMENTED : Invoker for gRPC not found"(consumer)/ "Invoker for gRPC not found"(provider)

Open
#16,397 9 comments 0 reactions 0 assignees View on GitHub
help wanted
Dominant language
Java
Stars
41.6k
Forks
26.4k
Avg merge
15h 13m
Merged PRs (30d)
4

Description

### Pre-check

- [x] I am sure that all the content I provide is in English.

### Search before asking

- [x] I had searched in the [issues](https://github.com/apache/dubbo/issues?q=is%3Aissue) and found no similar issues.

### Apache Dubbo Component

Java SDK (apache/dubbo)

### Dubbo Version

Dubbo 3.3.6, Triple protocol (`dubbo.protocol.name=tri`), instance mode , JDK 8.

### Steps to reproduce this issue

Graceful shutdown tears down the Triple `PathResolver` mapping *before* in-flight requests have drained.
A request that has already been accepted by the server, but has not yet resolved its invoker, will fail with
`UNIMPLEMENTED` once `afterUnExport` runs.

The race can be reproduced deterministically with two breakpoints on the **provider** side — no timing luck
required. Use a **thread-level** (suspend-thread, not suspend-all) breakpoint so the shutdown thread and the
request thread can be released independently.

Provider service (anything works, the body is irrelevant):

```java
@DubboService
public class DemoServiceImpl implements DemoService {
@Override
public String sayHello(String name) {
return "hello " + name;
}
}

Consumer: a single call is enough.

@DubboReference(timeout = 500000)
private DemoService demoService;

demoService.sayHello("dubbo");

Reproduction procedure (provider JVM, debugger attached):

1. Set a thread-suspending breakpoint at
org.apache.dubbo.rpc.protocol.tri.h12.grpc.GrpcRequestHandlerMapping#getRequestHandler, on the line

Invoker invoker = pathResolver.resolve(path.getPath(), group, version); // GrpcRequestHandlerMapping.java:62

2. Issue the consumer call. The request thread is now suspended before the path is resolved — i.e. the
request has already been accepted by a fully healthy provider.
3. Send TERM to the provider process to start graceful shutdown, with a second thread-suspending breakpoint at

pathResolver.unregister(invoker); // TripleProtocol.java:111, in TripleProtocol$1#afterUnExport

4. Step over that line so the unregister actually executes, then leave the shutdown thread suspended.
TriplePathResolver no longer contains the path mapping for the service.
5. Release the request thread from step 1 and step over line 62. resolve(...) now returns null, so control
falls into

if (invoker == null) {
throw notFound(); // GrpcRequestHandlerMapping.java:64
}

Result

- Provider log:
Invoker for gRPC not found
- Consumer:
org.apache.dubbo.rpc.RpcException: ... UNIMPLEMENTED : Invoker for gRPC not found

Without the debugger, the same thing happens naturally on every shutdown that overlaps with concurrent
traffic; the breakpoints only make the window deterministic.

### What you expected to happen

The request accepted in step 1 was accepted while the service was still exported, and the configured graceful
shutdown wait (dubbo.service.shutdown.wait) had not elapsed. It should have been served normally.

Concretely:

1. PathResolver.unregister(invoker) should not take effect until in-flight requests have drained — the path
mapping must outlive the request-draining phase rather than being removed at the start of it.
2. If the server has genuinely stopped accepting requests, the failure should be retriable: UNAVAILABLE,
or an HTTP/2 GOAWAY that lets the consumer re-route to another provider instance.

UNIMPLEMENTED is semantically wrong here — it means "this server does not implement the service", so
consumer-side cluster fault tolerance treats it as a permanent definition error and does not fail over. Every
rolling restart of a Triple provider therefore surfaces as business-visible errors on the consumer, which
defeats the purpose of graceful shutdown.

### Anything else

Code paths involved (3.3.6):

- org.apache.dubbo.rpc.protocol.tri.TripleProtocol$1#afterUnExport — pathResolver.unregister(invoker)
(TripleProtocol.java:111), registered by TripleProtocol#export
- org.apache.dubbo.rpc.protocol.tri.h12.grpc.GrpcRequestHandlerMapping#getRequestHandler
(GrpcRequestHandlerMapping.java:62 resolve, :64 throw notFound())
- org.apache.dubbo.rpc.PathResolver#unregister / #resolve,
org.apache.dubbo.rpc.protocol.tri.TriplePathResolver

The same window applies to any RequestHandlerMapping that resolves through PathResolver during shutdown,
not only the gRPC one.

### Do you have a (mini) reproduction demo?

- [x] Yes, I have a minimal reproduction demo to help resolve this issue more effectively!

### Are you willing to submit a pull request to fix on your own?

- [ ] Yes I am willing to submit a pull request on my own!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Research direction

Start with the deterministic debugger reproduction at GrpcRequestHandlerMapping#getRequestHandler, then read TripleProtocol$1#afterUnExport and the PathResolver/TriplePathResolver unregister and resolve paths. Compare the shutdown ordering around TripleProtocol.java:111 with the request lookup at GrpcRequestHandlerMapping.java:62-64. Done means accepted in-flight requests survive the configured graceful-shutdown wait, while genuinely rejected requests receive a retriable outcome.

Written by the indexing model from the issue text.

Assessment

Tech stack
grpc, java
Domain
backend-api-design, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.