[Java] Avro to arrow conversion fails for nested unions
- 主要言語
- Java
- スター
- 94
- フォーク
- 152
- 平均マージ
- 3日 16時間
- マージ済み PR(30日)
- 11
説明
### Describe the bug, including details regarding any error messages, version, and platform.
Hello arrow team,
I believe the Avro adapter fails on avro union types nested in records and arrays. The allocator tries to allocate increasingly more memory until it reaches the configured max (ie you can use `-Darrow.vector.max_allocation_bytes=8589934592` to tell the allocator to use 1GB and it will try to allocate all of it for an empty array field with a nested union type field).
`Memory required for vector capacity 508400 is (2097152), which is more than max allowed (1048576)`
I believe this is because the while loop [here](https://github.com/apache/arrow/blob/main/java/adapter/avro/src/main/java/org/apache/arrow/adapter/avro/consumers/AvroStructConsumer.java#L69-L75) never terminates as `UnionVector::getValueCapacity` always returns 0 when the union is nested in either a record or an array.
I have a branch [here](https://github.com/darcysaum-toast/arrow/tree/avro-nested-union) with a [failing test](https://github.com/darcysaum-toast/arrow/blob/avro-nested-union/java/adapter/avro/src/test/java/org/apache/arrow/adapter/avro/AvroToArrowIteratorTest.java#L171).
```
@Test
public void testNestedUnion() throws Exception {
Schema schema = getSchema("test_nested_union.avsc");
Schema topChildSchema = schema
.getField("f0")
.schema();
GenericRecord parent = new GenericData.Record(schema);
GenericRecord topChild = new GenericData.Record(topChildSchema);
topChild.put("fNestedUnion", 1);
parent.put("f0", topChild);
List roots = new ArrayList<>();
try (AvroToArrowVectorIterator iterator = convert(schema, singletonList(parent))) {
while (iterator.hasNext()) {
roots.add(iterator.next());
System.out.println(roots.get(roots.size() - 1).contentToTSVString());
}
}
}
```
The [schema](https://github.com/darcysaum-toast/arrow/blob/avro-nested-union/java/adapter/avro/src/test/resources/schema/test_nested_union.avsc) is as follows:
```
{
"namespace": "org.apache.arrow.avro",
"type": "record",
"name": "testArrayOfRecords",
"fields": [
{
"name": "f0",
"type": {
"type": "record",
"name": "NestedRecord",
"fields": [
{
"name": "fNestedUnion",
"type": ["null", "int"],
"default": null
}
]
}
}
]
```
Changing type of field `fNestedUnion` from `["null", "int"]` to `"int"` will make the test pass (and the iterator correctly prints out the arrow data).
### Component(s)
Java
コントリビューションガイド
調査の方向性
まず java/adapter/avro/src/main/java/org/apache/arrow/adapter/avro/consumers/AvroStructConsumer.java の 69~75 行付近にあるアロケーションループから始め、次に schema/test_nested_union.avsc を使用して AvroToArrowIteratorTest.java の失敗しているテストを実行します。ネストされた union の変換がアロケータを使い果たすことなく完了し、イテレータが期待される Arrow データを生成すれば完了です。
索引モデルが issue の本文から書いたものです。
評価
- 技術スタック
- java
- 領域
- data
- issue の種類
- バグ
- 難易度
- 3/5
- 見積もり時間
- 1〜2日
- 活発さ
- 停滞
- 明瞭さ
- おおむね明確
- 初心者へのやさしさ
- 45/100