[Java] Avro to Arrow Conversion fails for simple schema
- 主要語言
- Java
- 星號
- 94
- 分支
- 152
- 平均合併
- 3 天 16 小時
- 30 天內合併 PR
- 11
描述
### Describe the bug, including details regarding any error messages, version, and platform.
I am trying to use the [AvroToArrow](https://github.com/apache/arrow/blob/main/java/adapter/avro/src/main/java/org/apache/arrow/AvroToArrow.java) class to convert a dead simple avro record to arrow.
I am following this documentation: https://arrow.apache.org/cookbook/java/avro.html
Here is the simple avro schema `AvroToArrowExample.avsc`:
```
{
"namespace": "example",
"type": "record",
"name": "AvroToArrowExample",
"fields": [
{
"name": "id",
"type": "int"
}
]
}
```
I wrote a single record (see below) with the above schema to the file `sample_data.avro`:
```json
{"id": 1}
```
I copied the boilerplate code for performing avro to arrow conversion:
```java
// Copied code from arrow docs: https://arrow.apache.org/cookbook/java/avro.html
BinaryDecoder decoder = new DecoderFactory().binaryDecoder(new FileInputStream("sample_data.avro"), null);
Schema schema = new Schema.Parser().parse(new File("AvroToArrowExample.avsc"));
try (BufferAllocator allocator = new RootAllocator()) {
AvroToArrowConfig config = new AvroToArrowConfigBuilder(allocator).build();
try (AvroToArrowVectorIterator avroToArrowVectorIterator = AvroToArrow.avroToArrowIterator(schema, decoder, config)) {
while(avroToArrowVectorIterator.hasNext()) {
try (VectorSchemaRoot root = avroToArrowVectorIterator.next()) {
System.out.print(root.contentToTSVString());
}
}
}
}
```
This produces the following error:
```
java.lang.RuntimeException: Error occurs while getting next schema root.
at org.apache.arrow.AvroToArrowVectorIterator.next(AvroToArrowVectorIterator.java:172)
at [ My Project call to avroToArrowVectorIterator.next() ]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:108)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:40)
at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:60)
at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:52)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
at com.sun.proxy.$Proxy5.processTestClass(Unknown Source)
at org.gradle.api.internal.tasks.testing.worker.TestWorker$2.run(TestWorker.java:176)
at org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:129)
at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:100)
at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:60)
at org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56)
at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:113)
at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:65)
at worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69)
at worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)
Caused by: java.lang.RuntimeException: Error occurs while consuming data.
at org.apache.arrow.AvroToArrowVectorIterator.consumeData(AvroToArrowVectorIterator.java:128)
at org.apache.arrow.AvroToArrowVectorIterator.load(AvroToArrowVectorIterator.java:147)
at org.apache.arrow.AvroToArrowVectorIterator.next(AvroToArrowVectorIterator.java:169)
... 44 more
Caused by: java.lang.ArrayIndexOutOfBoundsException: arraycopy: length -1 is negative
at java.base/java.lang.System.arraycopy(Native Method)
at org.apache.avro.io.BinaryDecoder$ByteSource.compactAndFill(BinaryDecoder.java:748)
at org.apache.avro.io.BinaryDecoder.ensureBounds(BinaryDecoder.java:540)
at org.apache.avro.io.BinaryDecoder.readInt(BinaryDecoder.java:173)
at org.apache.arrow.consumers.AvroIntConsumer.consume(AvroIntConsumer.java:40)
at org.apache.arrow.consumers.CompositeAvroConsumer.consume(CompositeAvroConsumer.java:48)
at org.apache.arrow.AvroToArrowVectorIterator.consumeData(AvroToArrowVectorIterator.java:105)
... 46 more
Error occurs while consuming data.
java.lang.RuntimeException: Error occurs while consuming data.
at org.apache.arrow.AvroToArrowVectorIterator.consumeData(AvroToArrowVectorIterator.java:128)
at org.apache.arrow.AvroToArrowVectorIterator.load(AvroToArrowVectorIterator.java:147)
at org.apache.arrow.AvroToArrowVectorIterator.next(AvroToArrowVectorIterator.java:169)
at [ My Project call to avroToArrowVectorIterator.next() ]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:108)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:40)
at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:60)
at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:52)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
at com.sun.proxy.$Proxy5.processTestClass(Unknown Source)
at org.gradle.api.internal.tasks.testing.worker.TestWorker$2.run(TestWorker.java:176)
at org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:129)
at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:100)
at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:60)
at org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56)
at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:113)
at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:65)
at worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69)
at worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)
Caused by: java.lang.ArrayIndexOutOfBoundsException: arraycopy: length -1 is negative
at java.base/java.lang.System.arraycopy(Native Method)
at org.apache.avro.io.BinaryDecoder$ByteSource.compactAndFill(BinaryDecoder.java:748)
at org.apache.avro.io.BinaryDecoder.ensureBounds(BinaryDecoder.java:540)
at org.apache.avro.io.BinaryDecoder.readInt(BinaryDecoder.java:173)
at org.apache.arrow.consumers.AvroIntConsumer.consume(AvroIntConsumer.java:40)
at org.apache.arrow.consumers.CompositeAvroConsumer.consume(CompositeAvroConsumer.java:48)
at org.apache.arrow.AvroToArrowVectorIterator.consumeData(AvroToArrowVectorIterator.java:105)
... 46 more
```
### Component(s)
Java
貢獻指南
研究方向
使用回報的堆疊追蹤中的 AvroToArrow.java、cookbook 範例、AvroToArrowVectorIterator 和 AvroIntConsumer 重現此失敗。先追蹤從 AvroToArrowVectorIterator.next() 經過 consumeData() 的消費流程,並執行 Java 測試或進行聚焦的重現。提供的單筆記錄 Avro schema 能夠轉換且不出現回報的例外,即表示完成。
由索引模型根據 Issue 內容生成。
評估
- 技術堆疊
- java
- 領域
- data-engineering
- Issue 類型
- 缺陷
- 難度
- 3/5
- 預估耗時
- 1-2 天
- 活躍度
- 停滯
- 描述清晰度
- 需要釐清
- 新手友好度
- 35/100