[multistage][bug] error propagation during GRPC failure is not handled when setup is not finished.
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 189
Description
when GRPC internal failure occurs. the ReceivingMailbox doesn't propagate that error back to the next block fetcher if the message was the FIRST block
error from broker:
```
2023/11/01 08:45:52.204 ERROR [MultiStageBrokerRequestHandler] [jersey-server-managed-async-executor-7] Caught exception executing request 1879852788000000005: select * from tbl limit 6000,
Received error query execution result block: {250=ExecutionTimeoutError
ProcessingException(errorCode:250, message:ExecutionTimeoutError)
at org.apache.pinot.common.exception.QueryException.(QueryException.java:113)
at org.apache.pinot.common.response.broker.BrokerResponseNative.(BrokerResponseNative.java:53)
at org.apache.pinot.broker.requesthandler.MultiStageBrokerRequestHandler.handleRequest(MultiStageBrokerRequestHandler.java:156)
at org.apache.pinot.broker.requesthandler.BaseBrokerRequestHandler.handleRequest(BaseBrokerRequestHandler.java:279)}
org.apache.pinot.query.service.dispatch.QueryDispatcher.getResultTable(QueryDispatcher.java:266)
```
but the server log indicates that:
```
Nov 01, 2023 8:45:42 AM io.grpc.netty.shaded.io.grpc.netty.NettyServerStream$TransportState deframeFailed
WARNING: Exception processing message
io.grpc.StatusRuntimeException: RESOURCE_EXHAUSTED: gRPC message exceeds maximum size 16777216: 17562988
at io.grpc.Status.asRuntimeException(Status.java:530)
at io.grpc.internal.MessageDeframer.processHeader(MessageDeframer.java:392)
at io.grpc.internal.MessageDeframer.deliver(MessageDeframer.java:272)
at io.grpc.internal.MessageDeframer.deframe(MessageDeframer.java:178)
...
2023/11/01 08:45:42.788 ERROR [MailboxContentObserver] [grpc-default-executor-12] Got error before mailbox is set up
io.grpc.StatusRuntimeException: CANCELLED: client cancelled
at io.grpc.Status.asRuntimeException(Status.java:530) ~[grpc-api-1.53.0.jar:1.53.0]
at io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onCancel(ServerCalls.java:291) [grpc-stub-1.53.0.jar:1.53.0]
at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closedInternal(ServerCallImpl.java:378) [grpc-core-1.53.0.jar:1.53.0]
at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closed(ServerCallImpl.java:365) [grpc-core-1.53.0.jar:1.53.0]
at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed.runInContext(ServerImpl.java:923) [grpc-core-1.53.0.jar:1.53.0]
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) [grpc-core-1.53.0.jar:1.53.0]
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133) [grpc-core-1.53.0.jar:1.53.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
```
Contributor guide
Research direction
Start with ReceivingMailbox and MailboxContentObserver, focusing on the setup path when a GRPC failure arrives before the mailbox is initialized and the message is the first block. Trace how the error reaches the next block fetcher, using the MultiStageBrokerRequestHandler and QueryDispatcher stack traces for context. Done means the broker-side query error is propagated instead of being lost during initial setup.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- grpc, java
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 42/100