Multistage: connect(..) failed: Address family not supported by protocol release v1.0.0
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 195
Description
Pinot1.0.0 using multistage v2 option deployed from distribution, gets error running query.
Can see server/broker listening on the query and mailbox ports ok but client cannot connect to query.
Tried switching netty native v java implementation = no difference
Tried prefer ipv4 = no difference apart from the query showed that multistage was not enabled until the cluster config enable multistage was added in.
Online search suggests a shading issue with the grpc library in the distro.
https://stackoverflow.com/questions/73286776/grpc-unsupportedaddresstypeexception-but-only-when-packaged-with-shadowjar
```
QueryExecutionError:
Error dispatching query to server=yourserver.com@{10124,10126} stage=2
org.apache.pinot.query.service.dispatch.QueryDispatcher.submit(QueryDispatcher.java:144)
org.apache.pinot.query.service.dispatch.QueryDispatcher.submitAndReduce(QueryDispatcher.java:93)
org.apache.pinot.broker.requesthandler.MultiStageBrokerRequestHandler.handleRequest(MultiStageBrokerRequestHandler.java:179)
org.apache.pinot.broker.requesthandler.BaseBrokerRequestHandler.handleRequest(BaseBrokerRequestHandler.java:263)
UNAVAILABLE: io exception
io.grpc.Status.asRuntimeException(Status.java:539)
io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:487)
io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:576)
io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70)
connect(..) failed: Address family not supported by protocol: /yourserver.com:10124
connect(..) failed: Address family not supported by protocol
io.grpc.netty.shaded.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155)
io.grpc.netty.shaded.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128)
io.grpc.netty.shaded.io.netty.channel.unix.Socket.connect(Socket.java:313)
io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollChannel.doConnect0(AbstractEpollChannel.java:773)
connect(..) failed: Address family not supported by protocol
io.grpc.netty.shaded.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155)
io.grpc.netty.shaded.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128)
io.grpc.netty.shaded.io.netty.channel.unix.Socket.connect(Socket.java:313)
io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollChannel.doConnect0(AbstractEpollChannel.java:773)
```
Contributor guide
Research direction
Reproduce the failure with the Pinot 1.0.0 distribution, multistage v2, and the reported query, then start from QueryDispatcher.java:144 and the MultiStageBrokerRequestHandler.java:179 stack-trace entries. Inspect the packaged gRPC/Netty dependencies and compare them with the non-shaded implementation; done means multistage queries can connect to the query and mailbox ports without the address-family error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- grpc, java
- Domain
- distributed-systems, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 45/100