Arrow write: one bad value inside a ColumnGroup aborts the whole export, even in Mode.LOYAL
- Dominant language
- Kotlin
- Stars
- 1.1k
- Forks
- 83
- Avg merge
- 4d 12h
- Merged PRs (30d)
- 30
Description
One value that cannot be converted to the target Arrow type fails the entire write — no file is produced —
when it sits in a column inside a `ColumnGroup`. `Mode.LOYAL` is supposed to degrade such a value and report
it to `mismatchSubscriber` instead of aborting, and it does exactly that for a top-level column.
The reason is where the conversion happens: a `ColumnGroup` written into an `ArrowType.Struct` field has its
columns converted inside `ArrowWriterImpl.infillVector`, outside the `try/catch` in `allocateVectorAndInfill`
that applies `ArrowWriter.Mode` to a failed conversion. So the exception goes straight out of the writer and
the subscriber is never called.
```kotlin
val df = dataFrameOf(DataColumn.createColumnGroup("g", dataFrameOf("x")("abc", "def")))
val schema = Schema(listOf(Field("g", FieldType.nullable(ArrowType.Struct()),
listOf(Field("x", FieldType.nullable(ArrowType.Int(32, true)), null)))))
df.arrowWriter(schema, ArrowWriter.Mode.LOYAL) { println(it) }
.use { it.saveArrowFeatherToByteArray() }
// CellConversionException: Failed to convert 'abc' from kotlin.String to kotlin.Int in column 'x', row 0
// mismatchSubscriber: never called
```
Pre-existing and not specific to a vector type — the same happens for any child vector, e.g.
`is IntVector -> column.convertToInt()`. Noticed in #2063
(https://github.com/Kotlin/dataframe/pull/2063#discussion_r3981437295).
Contributor guide
Assessment
This issue has not been assessed yet.