Arrow 13.0.0: org.apache.arrow.flight.FlightRuntimeException: CANCELLED: Failed to stream message
- Dominant language
- Java
- Stars
- 94
- Forks
- 152
- Avg merge
- 3d 16h
- Merged PRs (30d)
- 11
Description
### Describe the bug, including details regarding any error messages, version, and platform.
Hello All,
I would like to ask your help in the following issue I am facing with.
I have a Java and a Python application in different pods running on Kubernetes. A running Arrow Flight server is on the Python side.
Arrow version: 13.0.0 (everything worked fine with version 12.0.1)
**flightClient.handshake();** throws this error and the method is in an infinite loop:
```BASH
org.apache.arrow.flight.FlightRuntimeException: CANCELLED: Failed to stream message
at org.apache.arrow.flight.CallStatus.toRuntimeException(CallStatus.java:131)
at org.apache.arrow.flight.grpc.StatusUtils.fromGrpcRuntimeException(StatusUtils.java:164)
at org.apache.arrow.flight.grpc.StatusUtils.fromThrowable(StatusUtils.java:185)
at org.apache.arrow.flight.auth2.ClientHandshakeWrapper.doClientHandshake(ClientHandshakeWrapper.java:59)
at org.apache.arrow.flight.FlightClient.handshake(FlightClient.java:212)
at ...
Caused by: io.grpc.StatusRuntimeException: INTERNAL: Failed to frame message
at io.grpc.Status.asRuntimeException(Status.java:530)
at io.grpc.internal.MessageFramer.writePayload(MessageFramer.java:154)
at io.grpc.internal.AbstractStream.writeMessage(AbstractStream.java:66)
at io.grpc.internal.ForwardingClientStream.writeMessage(ForwardingClientStream.java:37)
at io.grpc.internal.DelayedStream.writeMessage(DelayedStream.java:278)
at io.grpc.internal.RetriableStream$1SendMessageEntry.runWith(RetriableStream.java:567)
at io.grpc.internal.RetriableStream.delayOrExecute(RetriableStream.java:544)
at io.grpc.internal.RetriableStream.sendMessage(RetriableStream.java:575)
at io.grpc.internal.ClientCallImpl.sendMessageInternal(ClientCallImpl.java:525)
at io.grpc.internal.ClientCallImpl.sendMessage(ClientCallImpl.java:513)
at io.grpc.ForwardingClientCall.sendMessage(ForwardingClientCall.java:37)
at io.grpc.stub.ClientCalls$CallToStreamObserverAdapter.onNext(ClientCalls.java:378)
at org.apache.arrow.flight.auth2.ClientHandshakeWrapper.doClientHandshake(ClientHandshakeWrapper.java:48)
... 21 more
Caused by: java.lang.UnsupportedOperationException: This is supposed to be overridden by subclasses.
at com.google.protobuf.GeneratedMessageV3.getUnknownFields(GeneratedMessageV3.java:302)
at org.apache.arrow.flight.impl.Flight$HandshakeRequest.getSerializedSize(Flight.java:328)
at io.grpc.protobuf.lite.ProtoInputStream.available(ProtoInputStream.java:108)
at io.grpc.internal.MessageFramer.getKnownLength(MessageFramer.java:208)
at io.grpc.internal.MessageFramer.writePayload(MessageFramer.java:138)
... 32 more
```
### Component(s)
Java
Contributor guide
Research direction
Start at org.apache.arrow.flight.auth2.ClientHandshakeWrapper.doClientHandshake and the generated Flight.HandshakeRequest serialization shown in the stack trace. Compare the Java client behavior and dependencies between Arrow 12.0.1 and 13.0.0, reproducing the handshake between the Java and Python applications in Kubernetes. Done means flightClient.handshake() completes without the framing error or infinite retry loop.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- grpc, java, kubernetes, python
- Domain
- api, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100