[Bug] ArrowFormatCWriter will raise java.lang.IllegalArgumentException while writing arrow format with empty schema
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Paimon version
When we write rows to arrow format with empty paimon schema,following exception will raise
`java.lang.IllegalArgumentException
at org.apache.arrow.util.Preconditions.checkArgument(Preconditions.java:122)
at org.apache.arrow.vector.VectorSchemaRoot.getVector(VectorSchemaRoot.java:185)
at org.apache.paimon.arrow.ArrowUtils.serializeToCStruct(ArrowUtils.java:224)
at org.apache.paimon.arrow.vector.ArrowFormatCWriter.toCStruct(ArrowFormatCWriter.java:72)
at org.apache.paimon.arrow.vector.ArrowFormatWriterTest.testArrowFormatCWriterWithEmptySchema(ArrowFormatWriterTest.java:292)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)`
### Compute Engine
JavaAPI
### Minimal reproduce step
try (RootAllocator rootAllocator = new RootAllocator();
BufferAllocator allocator =
rootAllocator.newChildAllocator("paimonWriter", 0, Long.MAX_VALUE);
ArrowFormatCWriter writer =
new ArrowFormatCWriter(emptyschema, 4096, true, allocator)) {
for (int i = 0; i < 100; i++) {
writer.write(GenericRow.of());
}
writer.flush();
ArrowCStruct cStruct = writer.toCStruct();
Assertions.assertThat(cStruct).isNotNull();
writer.release();
}
### What doesn't meet your expectations?
It should work as normal.
### Anything else?
_No response_
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with ArrowFormatCWriter.toCStruct in ArrowFormatCWriter.java and the failing serialization path in ArrowUtils.java. Run ArrowFormatWriterTest.testArrowFormatCWriterWithEmptySchema using the empty-schema reproduction. Done means writing and flushing empty rows produces a non-null ArrowCStruct without IllegalArgumentException.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100