VectorAppender fails with OversizedAllocationException when appending to empty UnionVector
- Dominant language
- Java
- Stars
- 94
- Forks
- 152
- Avg merge
- 3d 16h
- Merged PRs (30d)
- 11
Description
### Describe the bug, including details regarding any error messages, version, and platform.
**Description:**
When appending a `UnionVector` to an empty target `UnionVector`, [this](https://github.com/apache/arrow-java/blob/main/vector/src/main/java/org/apache/arrow/vector/util/VectorAppender.java#L556) while loop runs infinitely until the exception occurs:
```
org.apache.arrow.vector.util.OversizedAllocationException: Unable to expand the buffer
```
It seems to be a issue with the `UnionVector.getValueCapacity()` method. Logging the capacity before and after calling `reAlloc()` shows that it's always `0`.
Debugging further, it seems like the value capacity of the `internalStruct` in the `UnionVector` always returns `0` because it has no children so [size](https://github.com/apache/arrow-java/blob/main/vector/src/main/java/org/apache/arrow/vector/complex/NonNullableStructVector.java#L373) is `0`.
```
@Override
public int getValueCapacity() {
return Math.min(getTypeBufferValueCapacity(), internalStruct.getValueCapacity());
}
```
**Steps to reproduce**
```
@Test
public void testAppendEmptyTargetUnionVector() {
final int length = 5;
try (final UnionVector target = UnionVector.empty("target", allocator);
final UnionVector delta = UnionVector.empty("delta", allocator)) {
// populate the delta vector
delta.setType(0, Types.MinorType.FLOAT4);
delta.setType(1, Types.MinorType.FLOAT4);
delta.setType(2, Types.MinorType.FLOAT4);
delta.setType(3, Types.MinorType.FLOAT4);
delta.setType(4, Types.MinorType.FLOAT4);
Float4Vector deltaFloatVector = delta.getFloat4Vector();
deltaFloatVector.allocateNew();
ValueVectorDataPopulator.setVector(deltaFloatVector, 1f, 2f, 3f, 4f, 5f);
assertEquals(length, deltaFloatVector.getValueCount());
delta.setValueCount(length);
VectorAppender appender = new VectorAppender(target);
delta.accept(appender, null);
assertEquals(length, target.getValueCount());
for (int i = 0; i < length; i++) {
Object floatObj = target.getObject(i);
assertTrue(floatObj instanceof Float);
assertEquals(i+1, ((Float) floatObj).intValue());
}
}
}
```
This issue does not occur if any value is set in the target vector before appending.
Contributor guide
Research direction
Start with the VectorAppender.java loop around line 556 and inspect UnionVector.getValueCapacity(), including NonNullableStructVector.java around line 373. Run the provided testAppendEmptyTargetUnionVector reproduction and compare capacity before and after reAlloc(). Done means appending to an empty UnionVector completes without OversizedAllocationException and preserves all five float values.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 55/100