Java - Flight SQL - `StackOverflowException` caused by `org.apache.arrow.flight.FlightStream.next`
- Lenguaje dominante
- Java
- Estrellas
- 94
- Forks
- 152
- Merge medio
- 3 d 16 h
- PR fusionados (30 d)
- 11
Descripción
### Describe the bug, including details regarding any error messages, version, and platform.
We're experimenting with Arrow Flight SQL and wrote an initial basic prototype server.
I've tried hitting it with the Java library, but it's a little buggy so far.
The data set I'm querying is a few GBs of data across 10'000'000 rows and 21 columns.
There are 10 double columns and 10 symbol (`Dictionary`) columns.
See: https://github.com/timescale/tsbs to get an idea of the type of data being queried.
I think the Java Flight SQL client is struggling with the response sent back.
```
java.lang.StackOverflowError
at java.base/java.util.Spliterator.getExactSizeIfKnown(Spliterator.java:414)
at java.base/java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:526)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:513)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.base/java.util.stream.FindOps$FindOp.evaluateSequential(FindOps.java:150)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.IntPipeline.findFirst(IntPipeline.java:552)
at java.base/java.text.DecimalFormatSymbols.findNonFormatChar(DecimalFormatSymbols.java:844)
at java.base/java.text.DecimalFormatSymbols.initialize(DecimalFormatSymbols.java:815)
at java.base/java.text.DecimalFormatSymbols.(DecimalFormatSymbols.java:115)
at java.base/sun.util.locale.provider.DecimalFormatSymbolsProviderImpl.getInstance(DecimalFormatSymbolsProviderImpl.java:85)
at java.base/java.text.DecimalFormatSymbols.getInstance(DecimalFormatSymbols.java:182)
at java.base/java.util.Formatter.zero(Formatter.java:2450)
at java.base/java.util.Formatter$FormatSpecifier.getZero(Formatter.java:4450)
at java.base/java.util.Formatter$FormatSpecifier.localizedMagnitude(Formatter.java:4466)
at java.base/java.util.Formatter$FormatSpecifier.print(Formatter.java:3276)
at java.base/java.util.Formatter$FormatSpecifier.print(Formatter.java:3261)
at java.base/java.util.Formatter$FormatSpecifier.printInteger(Formatter.java:2957)
at java.base/java.util.Formatter$FormatSpecifier.print(Formatter.java:2918)
at java.base/java.util.Formatter.format(Formatter.java:2689)
at java.base/java.util.Formatter.format(Formatter.java:2625)
at java.base/java.lang.String.format(String.java:4141)
at org.apache.arrow.memory.util.HistoricalLog.recordEvent(HistoricalLog.java:82)
at org.apache.arrow.memory.BufferLedger.retain(BufferLedger.java:182)
at org.apache.arrow.memory.BufferLedger.retain(BufferLedger.java:169)
at org.apache.arrow.vector.ipc.message.ArrowRecordBatch.(ArrowRecordBatch.java:92)
at org.apache.arrow.vector.ipc.message.ArrowRecordBatch.(ArrowRecordBatch.java:69)
at org.apache.arrow.vector.ipc.message.MessageSerializer.deserializeRecordBatch(MessageSerializer.java:438)
at org.apache.arrow.vector.ipc.message.MessageSerializer.deserializeDictionaryBatch(MessageSerializer.java:514)
at org.apache.arrow.vector.ipc.message.MessageSerializer.deserializeDictionaryBatch(MessageSerializer.java:529)
at org.apache.arrow.flight.ArrowMessage.asDictionaryBatch(ArrowMessage.java:273)
at org.apache.arrow.flight.FlightStream.next(FlightStream.java:264)
at org.apache.arrow.flight.FlightStream.next(FlightStream.java:280)
at org.apache.arrow.flight.FlightStream.next(FlightStream.java:280)
at org.apache.arrow.flight.FlightStream.next(FlightStream.java:280)
at org.apache.arrow.flight.FlightStream.next(FlightStream.java:280)
at org.apache.arrow.flight.FlightStream.next(FlightStream.java:280)
at org.apache.arrow.flight.FlightStream.next(FlightStream.java:280)
at org.apache.arrow.flight.FlightStream.next(FlightStream.java:280)
at org.apache.arrow.flight.FlightStream.next(FlightStream.java:280)
at org.apache.arrow.flight.FlightStream.next(FlightStream.java:280)
at org.apache.arrow.flight.FlightStream.next(FlightStream.java:280)
at org.apache.arrow.flight.FlightStream.next(FlightStream.java:280)
at org.apache.arrow.flight.FlightStream.next(FlightStream.java:280)
at org.apache.arrow.flight.FlightStream.next(FlightStream.java:280)
at org.apache.arrow.flight.FlightStream.next(FlightStream.java:280)
at org.apache.arrow.flight.FlightStream.next(FlightStream.java:280)
at org.apache.arrow.flight.FlightStream.next(FlightStream.java:280)
at org.apache.arrow.flight.FlightStream.next(FlightStream.java:280)
at org.apache.arrow.flight.FlightStream.next(FlightStream.java:280)
at org.apache.arrow.flight.FlightStream.next(FlightStream.java:280)
at org.apache.arrow.flight.FlightStream.next(FlightStream.java:280)
at org.apache.arrow.flight.FlightStream.next(FlightStream.java:280)
....
```
Looking at `FlightStream.java:280`, there's a recursive call to `next()`.
```java
public boolean next() {
...
} else if (msg.getMessageType() == HeaderType.DICTIONARY_BATCH) {
...
return next(); // <------------- culprit
...
}
```
This should be fixed to allow querying large datasets.
Before I forget, here's the version I'm using:
```
org.apache.arrow
flight-sql
11.0.0
```
### Component(s)
Java
Guía de contribución
Evaluación
Este issue todavía no se ha evaluado.