[Java] Avro to Arrow Conversion fails for simple schema
- Lenguaje dominante
- Java
- Estrellas
- 94
- Forks
- 152
- Merge medio
- 3 d 16 h
- PR fusionados (30 d)
- 11
Descripción
### Describe the bug, including details regarding any error messages, version, and platform.
I am trying to use the [AvroToArrow](https://github.com/apache/arrow/blob/main/java/adapter/avro/src/main/java/org/apache/arrow/AvroToArrow.java) class to convert a dead simple avro record to arrow.
I am following this documentation: https://arrow.apache.org/cookbook/java/avro.html
Here is the simple avro schema `AvroToArrowExample.avsc`:
```
{
"namespace": "example",
"type": "record",
"name": "AvroToArrowExample",
"fields": [
{
"name": "id",
"type": "int"
}
]
}
```
I wrote a single record (see below) with the above schema to the file `sample_data.avro`:
```json
{"id": 1}
```
I copied the boilerplate code for performing avro to arrow conversion:
```java
// Copied code from arrow docs: https://arrow.apache.org/cookbook/java/avro.html
BinaryDecoder decoder = new DecoderFactory().binaryDecoder(new FileInputStream("sample_data.avro"), null);
Schema schema = new Schema.Parser().parse(new File("AvroToArrowExample.avsc"));
try (BufferAllocator allocator = new RootAllocator()) {
AvroToArrowConfig config = new AvroToArrowConfigBuilder(allocator).build();
try (AvroToArrowVectorIterator avroToArrowVectorIterator = AvroToArrow.avroToArrowIterator(schema, decoder, config)) {
while(avroToArrowVectorIterator.hasNext()) {
try (VectorSchemaRoot root = avroToArrowVectorIterator.next()) {
System.out.print(root.contentToTSVString());
}
}
}
}
```
This produces the following error:
```
java.lang.RuntimeException: Error occurs while getting next schema root.
at org.apache.arrow.AvroToArrowVectorIterator.next(AvroToArrowVectorIterator.java:172)
at [ My Project call to avroToArrowVectorIterator.next() ]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:108)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:40)
at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:60)
at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:52)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
at com.sun.proxy.$Proxy5.processTestClass(Unknown Source)
at org.gradle.api.internal.tasks.testing.worker.TestWorker$2.run(TestWorker.java:176)
at org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:129)
at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:100)
at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:60)
at org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56)
at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:113)
at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:65)
at worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69)
at worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)
Caused by: java.lang.RuntimeException: Error occurs while consuming data.
at org.apache.arrow.AvroToArrowVectorIterator.consumeData(AvroToArrowVectorIterator.java:128)
at org.apache.arrow.AvroToArrowVectorIterator.load(AvroToArrowVectorIterator.java:147)
at org.apache.arrow.AvroToArrowVectorIterator.next(AvroToArrowVectorIterator.java:169)
... 44 more
Caused by: java.lang.ArrayIndexOutOfBoundsException: arraycopy: length -1 is negative
at java.base/java.lang.System.arraycopy(Native Method)
at org.apache.avro.io.BinaryDecoder$ByteSource.compactAndFill(BinaryDecoder.java:748)
at org.apache.avro.io.BinaryDecoder.ensureBounds(BinaryDecoder.java:540)
at org.apache.avro.io.BinaryDecoder.readInt(BinaryDecoder.java:173)
at org.apache.arrow.consumers.AvroIntConsumer.consume(AvroIntConsumer.java:40)
at org.apache.arrow.consumers.CompositeAvroConsumer.consume(CompositeAvroConsumer.java:48)
at org.apache.arrow.AvroToArrowVectorIterator.consumeData(AvroToArrowVectorIterator.java:105)
... 46 more
Error occurs while consuming data.
java.lang.RuntimeException: Error occurs while consuming data.
at org.apache.arrow.AvroToArrowVectorIterator.consumeData(AvroToArrowVectorIterator.java:128)
at org.apache.arrow.AvroToArrowVectorIterator.load(AvroToArrowVectorIterator.java:147)
at org.apache.arrow.AvroToArrowVectorIterator.next(AvroToArrowVectorIterator.java:169)
at [ My Project call to avroToArrowVectorIterator.next() ]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:108)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:40)
at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:60)
at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:52)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
at com.sun.proxy.$Proxy5.processTestClass(Unknown Source)
at org.gradle.api.internal.tasks.testing.worker.TestWorker$2.run(TestWorker.java:176)
at org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:129)
at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:100)
at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:60)
at org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56)
at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:113)
at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:65)
at worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69)
at worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)
Caused by: java.lang.ArrayIndexOutOfBoundsException: arraycopy: length -1 is negative
at java.base/java.lang.System.arraycopy(Native Method)
at org.apache.avro.io.BinaryDecoder$ByteSource.compactAndFill(BinaryDecoder.java:748)
at org.apache.avro.io.BinaryDecoder.ensureBounds(BinaryDecoder.java:540)
at org.apache.avro.io.BinaryDecoder.readInt(BinaryDecoder.java:173)
at org.apache.arrow.consumers.AvroIntConsumer.consume(AvroIntConsumer.java:40)
at org.apache.arrow.consumers.CompositeAvroConsumer.consume(CompositeAvroConsumer.java:48)
at org.apache.arrow.AvroToArrowVectorIterator.consumeData(AvroToArrowVectorIterator.java:105)
... 46 more
```
### Component(s)
Java
Guía de contribución
Línea de trabajo
Reproduce el fallo utilizando AvroToArrow.java, el ejemplo del cookbook, AvroToArrowVectorIterator y AvroIntConsumer de la traza de pila indicada. Empieza siguiendo el consumo desde AvroToArrowVectorIterator.next() a través de consumeData() y ejecuta las pruebas de Java o una reproducción enfocada. Se considera terminado cuando el esquema de Avro de un solo registro proporcionado se convierte sin la excepción indicada.
Escrito por el modelo de indexación a partir del texto del issue.
Evaluación
- Stack tecnológico
- java
- Área
- data-engineering
- Tipo de issue
- Error
- Dificultad
- 3/5
- Tiempo estimado
- 1-2 días
- Estado de actividad
- Estancado
- Claridad
- Necesita aclaración
- Aptitud para principiantes
- 35/100