apache / apache/arrow-java

Java - Flight SQL - `StackOverflowException` caused by `org.apache.arrow.flight.FlightStream.next`

Open
#172 5 comments 0 reactions 1 assignee Claimed by @vibhatha View on GitHub
Dominant language
Java
Stars
94
Forks
152
Avg merge
3d 16h
Merged PRs (30d)
11

Description

### Describe the bug, including details regarding any error messages, version, and platform.

We're experimenting with Arrow Flight SQL and wrote an initial basic prototype server.

I've tried hitting it with the Java library, but it's a little buggy so far.

The data set I'm querying is a few GBs of data across 10'000'000 rows and 21 columns.
There are 10 double columns and 10 symbol (`Dictionary`) columns.

See: https://github.com/timescale/tsbs to get an idea of the type of data being queried.

I think the Java Flight SQL client is struggling with the response sent back.

```
java.lang.StackOverflowError
at java.base/java.util.Spliterator.getExactSizeIfKnown(Spliterator.java:414)
at java.base/java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:526)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:513)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.base/java.util.stream.FindOps$FindOp.evaluateSequential(FindOps.java:150)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.IntPipeline.findFirst(IntPipeline.java:552)
at java.base/java.text.DecimalFormatSymbols.findNonFormatChar(DecimalFormatSymbols.java:844)
at java.base/java.text.DecimalFormatSymbols.initialize(DecimalFormatSymbols.java:815)
at java.base/java.text.DecimalFormatSymbols.(DecimalFormatSymbols.java:115)
at java.base/sun.util.locale.provider.DecimalFormatSymbolsProviderImpl.getInstance(DecimalFormatSymbolsProviderImpl.java:85)
at java.base/java.text.DecimalFormatSymbols.getInstance(DecimalFormatSymbols.java:182)
at java.base/java.util.Formatter.zero(Formatter.java:2450)
at java.base/java.util.Formatter$FormatSpecifier.getZero(Formatter.java:4450)
at java.base/java.util.Formatter$FormatSpecifier.localizedMagnitude(Formatter.java:4466)
at java.base/java.util.Formatter$FormatSpecifier.print(Formatter.java:3276)
at java.base/java.util.Formatter$FormatSpecifier.print(Formatter.java:3261)
at java.base/java.util.Formatter$FormatSpecifier.printInteger(Formatter.java:2957)
at java.base/java.util.Formatter$FormatSpecifier.print(Formatter.java:2918)
at java.base/java.util.Formatter.format(Formatter.java:2689)
at java.base/java.util.Formatter.format(Formatter.java:2625)
at java.base/java.lang.String.format(String.java:4141)
at org.apache.arrow.memory.util.HistoricalLog.recordEvent(HistoricalLog.java:82)
at org.apache.arrow.memory.BufferLedger.retain(BufferLedger.java:182)
at org.apache.arrow.memory.BufferLedger.retain(BufferLedger.java:169)
at org.apache.arrow.vector.ipc.message.ArrowRecordBatch.(ArrowRecordBatch.java:92)
at org.apache.arrow.vector.ipc.message.ArrowRecordBatch.(ArrowRecordBatch.java:69)
at org.apache.arrow.vector.ipc.message.MessageSerializer.deserializeRecordBatch(MessageSerializer.java:438)
at org.apache.arrow.vector.ipc.message.MessageSerializer.deserializeDictionaryBatch(MessageSerializer.java:514)
at org.apache.arrow.vector.ipc.message.MessageSerializer.deserializeDictionaryBatch(MessageSerializer.java:529)
at org.apache.arrow.flight.ArrowMessage.asDictionaryBatch(ArrowMessage.java:273)
at org.apache.arrow.flight.FlightStream.next(FlightStream.java:264)
at org.apache.arrow.flight.FlightStream.next(FlightStream.java:280)
at org.apache.arrow.flight.FlightStream.next(FlightStream.java:280)
at org.apache.arrow.flight.FlightStream.next(FlightStream.java:280)
at org.apache.arrow.flight.FlightStream.next(FlightStream.java:280)
at org.apache.arrow.flight.FlightStream.next(FlightStream.java:280)
at org.apache.arrow.flight.FlightStream.next(FlightStream.java:280)
at org.apache.arrow.flight.FlightStream.next(FlightStream.java:280)
at org.apache.arrow.flight.FlightStream.next(FlightStream.java:280)
at org.apache.arrow.flight.FlightStream.next(FlightStream.java:280)
at org.apache.arrow.flight.FlightStream.next(FlightStream.java:280)
at org.apache.arrow.flight.FlightStream.next(FlightStream.java:280)
at org.apache.arrow.flight.FlightStream.next(FlightStream.java:280)
at org.apache.arrow.flight.FlightStream.next(FlightStream.java:280)
at org.apache.arrow.flight.FlightStream.next(FlightStream.java:280)
at org.apache.arrow.flight.FlightStream.next(FlightStream.java:280)
at org.apache.arrow.flight.FlightStream.next(FlightStream.java:280)
at org.apache.arrow.flight.FlightStream.next(FlightStream.java:280)
at org.apache.arrow.flight.FlightStream.next(FlightStream.java:280)
at org.apache.arrow.flight.FlightStream.next(FlightStream.java:280)
at org.apache.arrow.flight.FlightStream.next(FlightStream.java:280)
at org.apache.arrow.flight.FlightStream.next(FlightStream.java:280)
....
```

Looking at `FlightStream.java:280`, there's a recursive call to `next()`.

```java
public boolean next() {
...
} else if (msg.getMessageType() == HeaderType.DICTIONARY_BATCH) {
...
return next(); // <------------- culprit
...
}
```

This should be fixed to allow querying large datasets.

Before I forget, here's the version I'm using:

```

org.apache.arrow
flight-sql
11.0.0

```

### Component(s)

Java

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.