apache / apache/pinot

[multistage][bug] error propagation during GRPC failure is not handled when setup is not finished.

Closed
#11,919 1 comment 0 reactions 0 assignees View on GitHub
bug multi-stage stale
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

when GRPC internal failure occurs. the ReceivingMailbox doesn't propagate that error back to the next block fetcher if the message was the FIRST block
error from broker:
```
2023/11/01 08:45:52.204 ERROR [MultiStageBrokerRequestHandler] [jersey-server-managed-async-executor-7] Caught exception executing request 1879852788000000005: select * from tbl limit 6000,
Received error query execution result block: {250=ExecutionTimeoutError
ProcessingException(errorCode:250, message:ExecutionTimeoutError)
at org.apache.pinot.common.exception.QueryException.(QueryException.java:113)
at org.apache.pinot.common.response.broker.BrokerResponseNative.(BrokerResponseNative.java:53)
at org.apache.pinot.broker.requesthandler.MultiStageBrokerRequestHandler.handleRequest(MultiStageBrokerRequestHandler.java:156)
at org.apache.pinot.broker.requesthandler.BaseBrokerRequestHandler.handleRequest(BaseBrokerRequestHandler.java:279)}
org.apache.pinot.query.service.dispatch.QueryDispatcher.getResultTable(QueryDispatcher.java:266)
```

but the server log indicates that:
```
Nov 01, 2023 8:45:42 AM io.grpc.netty.shaded.io.grpc.netty.NettyServerStream$TransportState deframeFailed
WARNING: Exception processing message
io.grpc.StatusRuntimeException: RESOURCE_EXHAUSTED: gRPC message exceeds maximum size 16777216: 17562988
at io.grpc.Status.asRuntimeException(Status.java:530)
at io.grpc.internal.MessageDeframer.processHeader(MessageDeframer.java:392)
at io.grpc.internal.MessageDeframer.deliver(MessageDeframer.java:272)
at io.grpc.internal.MessageDeframer.deframe(MessageDeframer.java:178)
...
2023/11/01 08:45:42.788 ERROR [MailboxContentObserver] [grpc-default-executor-12] Got error before mailbox is set up
io.grpc.StatusRuntimeException: CANCELLED: client cancelled
at io.grpc.Status.asRuntimeException(Status.java:530) ~[grpc-api-1.53.0.jar:1.53.0]
at io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onCancel(ServerCalls.java:291) [grpc-stub-1.53.0.jar:1.53.0]
at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closedInternal(ServerCallImpl.java:378) [grpc-core-1.53.0.jar:1.53.0]
at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closed(ServerCallImpl.java:365) [grpc-core-1.53.0.jar:1.53.0]
at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed.runInContext(ServerImpl.java:923) [grpc-core-1.53.0.jar:1.53.0]
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) [grpc-core-1.53.0.jar:1.53.0]
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133) [grpc-core-1.53.0.jar:1.53.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
```

Contributor guide

Open the contributing guide

Research direction

Start with ReceivingMailbox and MailboxContentObserver, focusing on the setup path when a GRPC failure arrives before the mailbox is initialized and the message is the first block. Trace how the error reaches the next block fetcher, using the MultiStageBrokerRequestHandler and QueryDispatcher stack traces for context. Done means the broker-side query error is propagated instead of being lost during initial setup.

Written by the indexing model from the issue text.

Assessment

Tech stack
grpc, java
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.