[Java] Avro to arrow conversion fails for nested unions
- 主要語言
- Java
- 星號
- 94
- 分支
- 152
- 平均合併
- 3 天 16 小時
- 30 天內合併 PR
- 11
描述
### Describe the bug, including details regarding any error messages, version, and platform.
Hello arrow team,
I believe the Avro adapter fails on avro union types nested in records and arrays. The allocator tries to allocate increasingly more memory until it reaches the configured max (ie you can use `-Darrow.vector.max_allocation_bytes=8589934592` to tell the allocator to use 1GB and it will try to allocate all of it for an empty array field with a nested union type field).
`Memory required for vector capacity 508400 is (2097152), which is more than max allowed (1048576)`
I believe this is because the while loop [here](https://github.com/apache/arrow/blob/main/java/adapter/avro/src/main/java/org/apache/arrow/adapter/avro/consumers/AvroStructConsumer.java#L69-L75) never terminates as `UnionVector::getValueCapacity` always returns 0 when the union is nested in either a record or an array.
I have a branch [here](https://github.com/darcysaum-toast/arrow/tree/avro-nested-union) with a [failing test](https://github.com/darcysaum-toast/arrow/blob/avro-nested-union/java/adapter/avro/src/test/java/org/apache/arrow/adapter/avro/AvroToArrowIteratorTest.java#L171).
```
@Test
public void testNestedUnion() throws Exception {
Schema schema = getSchema("test_nested_union.avsc");
Schema topChildSchema = schema
.getField("f0")
.schema();
GenericRecord parent = new GenericData.Record(schema);
GenericRecord topChild = new GenericData.Record(topChildSchema);
topChild.put("fNestedUnion", 1);
parent.put("f0", topChild);
List roots = new ArrayList<>();
try (AvroToArrowVectorIterator iterator = convert(schema, singletonList(parent))) {
while (iterator.hasNext()) {
roots.add(iterator.next());
System.out.println(roots.get(roots.size() - 1).contentToTSVString());
}
}
}
```
The [schema](https://github.com/darcysaum-toast/arrow/blob/avro-nested-union/java/adapter/avro/src/test/resources/schema/test_nested_union.avsc) is as follows:
```
{
"namespace": "org.apache.arrow.avro",
"type": "record",
"name": "testArrayOfRecords",
"fields": [
{
"name": "f0",
"type": {
"type": "record",
"name": "NestedRecord",
"fields": [
{
"name": "fNestedUnion",
"type": ["null", "int"],
"default": null
}
]
}
}
]
```
Changing type of field `fNestedUnion` from `["null", "int"]` to `"int"` will make the test pass (and the iterator correctly prints out the arrow data).
### Component(s)
Java
貢獻指南
研究方向
先從 java/adapter/avro/src/main/java/org/apache/arrow/adapter/avro/consumers/AvroStructConsumer.java 中第 69-75 行附近的配置循環開始,然後使用 schema/test_nested_union.avsc 執行 AvroToArrowIteratorTest.java 中失敗的測試。完成的標準是巢狀 union 轉換在不耗盡配置器的情況下完成,且迭代器產生預期的 Arrow 資料。
由索引模型根據 Issue 內容生成。
評估
- 技術堆疊
- java
- 領域
- data
- Issue 類型
- 缺陷
- 難度
- 3/5
- 預估耗時
- 1-2 天
- 活躍度
- 停滯
- 描述清晰度
- 基本清楚
- 新手友好度
- 45/100