[Java] Avro to Arrow Conversion fails for simple schema
- Langage dominant
- Java
- Étoiles
- 94
- Forks
- 152
- Merge moyen
- 3 j 16 h
- PR mergées (30 j)
- 11
Description
### Describe the bug, including details regarding any error messages, version, and platform.
I am trying to use the [AvroToArrow](https://github.com/apache/arrow/blob/main/java/adapter/avro/src/main/java/org/apache/arrow/AvroToArrow.java) class to convert a dead simple avro record to arrow.
I am following this documentation: https://arrow.apache.org/cookbook/java/avro.html
Here is the simple avro schema `AvroToArrowExample.avsc`:
```
{
"namespace": "example",
"type": "record",
"name": "AvroToArrowExample",
"fields": [
{
"name": "id",
"type": "int"
}
]
}
```
I wrote a single record (see below) with the above schema to the file `sample_data.avro`:
```json
{"id": 1}
```
I copied the boilerplate code for performing avro to arrow conversion:
```java
// Copied code from arrow docs: https://arrow.apache.org/cookbook/java/avro.html
BinaryDecoder decoder = new DecoderFactory().binaryDecoder(new FileInputStream("sample_data.avro"), null);
Schema schema = new Schema.Parser().parse(new File("AvroToArrowExample.avsc"));
try (BufferAllocator allocator = new RootAllocator()) {
AvroToArrowConfig config = new AvroToArrowConfigBuilder(allocator).build();
try (AvroToArrowVectorIterator avroToArrowVectorIterator = AvroToArrow.avroToArrowIterator(schema, decoder, config)) {
while(avroToArrowVectorIterator.hasNext()) {
try (VectorSchemaRoot root = avroToArrowVectorIterator.next()) {
System.out.print(root.contentToTSVString());
}
}
}
}
```
This produces the following error:
```
java.lang.RuntimeException: Error occurs while getting next schema root.
at org.apache.arrow.AvroToArrowVectorIterator.next(AvroToArrowVectorIterator.java:172)
at [ My Project call to avroToArrowVectorIterator.next() ]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:108)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:40)
at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:60)
at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:52)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
at com.sun.proxy.$Proxy5.processTestClass(Unknown Source)
at org.gradle.api.internal.tasks.testing.worker.TestWorker$2.run(TestWorker.java:176)
at org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:129)
at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:100)
at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:60)
at org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56)
at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:113)
at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:65)
at worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69)
at worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)
Caused by: java.lang.RuntimeException: Error occurs while consuming data.
at org.apache.arrow.AvroToArrowVectorIterator.consumeData(AvroToArrowVectorIterator.java:128)
at org.apache.arrow.AvroToArrowVectorIterator.load(AvroToArrowVectorIterator.java:147)
at org.apache.arrow.AvroToArrowVectorIterator.next(AvroToArrowVectorIterator.java:169)
... 44 more
Caused by: java.lang.ArrayIndexOutOfBoundsException: arraycopy: length -1 is negative
at java.base/java.lang.System.arraycopy(Native Method)
at org.apache.avro.io.BinaryDecoder$ByteSource.compactAndFill(BinaryDecoder.java:748)
at org.apache.avro.io.BinaryDecoder.ensureBounds(BinaryDecoder.java:540)
at org.apache.avro.io.BinaryDecoder.readInt(BinaryDecoder.java:173)
at org.apache.arrow.consumers.AvroIntConsumer.consume(AvroIntConsumer.java:40)
at org.apache.arrow.consumers.CompositeAvroConsumer.consume(CompositeAvroConsumer.java:48)
at org.apache.arrow.AvroToArrowVectorIterator.consumeData(AvroToArrowVectorIterator.java:105)
... 46 more
Error occurs while consuming data.
java.lang.RuntimeException: Error occurs while consuming data.
at org.apache.arrow.AvroToArrowVectorIterator.consumeData(AvroToArrowVectorIterator.java:128)
at org.apache.arrow.AvroToArrowVectorIterator.load(AvroToArrowVectorIterator.java:147)
at org.apache.arrow.AvroToArrowVectorIterator.next(AvroToArrowVectorIterator.java:169)
at [ My Project call to avroToArrowVectorIterator.next() ]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:108)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:40)
at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:60)
at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:52)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
at com.sun.proxy.$Proxy5.processTestClass(Unknown Source)
at org.gradle.api.internal.tasks.testing.worker.TestWorker$2.run(TestWorker.java:176)
at org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:129)
at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:100)
at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:60)
at org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56)
at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:113)
at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:65)
at worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69)
at worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)
Caused by: java.lang.ArrayIndexOutOfBoundsException: arraycopy: length -1 is negative
at java.base/java.lang.System.arraycopy(Native Method)
at org.apache.avro.io.BinaryDecoder$ByteSource.compactAndFill(BinaryDecoder.java:748)
at org.apache.avro.io.BinaryDecoder.ensureBounds(BinaryDecoder.java:540)
at org.apache.avro.io.BinaryDecoder.readInt(BinaryDecoder.java:173)
at org.apache.arrow.consumers.AvroIntConsumer.consume(AvroIntConsumer.java:40)
at org.apache.arrow.consumers.CompositeAvroConsumer.consume(CompositeAvroConsumer.java:48)
at org.apache.arrow.AvroToArrowVectorIterator.consumeData(AvroToArrowVectorIterator.java:105)
... 46 more
```
### Component(s)
Java
Guide de contribution
Ouvrir le guide de contribution
Piste de recherche
Reproduisez l’échec à l’aide de AvroToArrow.java, de l’exemple du cookbook, de AvroToArrowVectorIterator et de AvroIntConsumer mentionnés dans la stack trace fournie. Commencez par suivre la consommation depuis AvroToArrowVectorIterator.next() via consumeData(), puis exécutez les tests Java ou une reproduction ciblée. C’est terminé lorsque le schéma Avro à enregistrement unique fourni est converti sans l’exception signalée.
Rédigé par le modèle d'indexation à partir du texte de l'issue.
Évaluation
- Stack technique
- java
- Domaine
- data-engineering
- Type d'issue
- Bug
- Difficulté
- 3/5
- Temps estimé
- 1-2 jours
- Activité
- À l'abandon
- Clarté
- À clarifier
- Accessibilité débutants
- 35/100