apache / apache/paimon

[Bug] ArrowFormatCWriter will raise java.lang.IllegalArgumentException while writing arrow format with empty schema

Open
#6,452 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Paimon version

When we write rows to arrow format with empty paimon schema,following exception will raise
`java.lang.IllegalArgumentException
at org.apache.arrow.util.Preconditions.checkArgument(Preconditions.java:122)
at org.apache.arrow.vector.VectorSchemaRoot.getVector(VectorSchemaRoot.java:185)
at org.apache.paimon.arrow.ArrowUtils.serializeToCStruct(ArrowUtils.java:224)
at org.apache.paimon.arrow.vector.ArrowFormatCWriter.toCStruct(ArrowFormatCWriter.java:72)
at org.apache.paimon.arrow.vector.ArrowFormatWriterTest.testArrowFormatCWriterWithEmptySchema(ArrowFormatWriterTest.java:292)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)`

### Compute Engine

JavaAPI

### Minimal reproduce step

try (RootAllocator rootAllocator = new RootAllocator();
BufferAllocator allocator =
rootAllocator.newChildAllocator("paimonWriter", 0, Long.MAX_VALUE);
ArrowFormatCWriter writer =
new ArrowFormatCWriter(emptyschema, 4096, true, allocator)) {
for (int i = 0; i < 100; i++) {
writer.write(GenericRow.of());
}
writer.flush();
ArrowCStruct cStruct = writer.toCStruct();
Assertions.assertThat(cStruct).isNotNull();
writer.release();
}

### What doesn't meet your expectations?

It should work as normal.

### Anything else?

_No response_

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with ArrowFormatCWriter.toCStruct in ArrowFormatCWriter.java and the failing serialization path in ArrowUtils.java. Run ArrowFormatWriterTest.testArrowFormatCWriterWithEmptySchema using the empty-schema reproduction. Done means writing and flushing empty rows produces a non-null ArrowCStruct without IllegalArgumentException.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.