Kotlin / Kotlin/dataframe

Arrow write: one bad value inside a ColumnGroup aborts the whole export, even in Mode.LOYAL

Open
#2,075 0 comments 0 reactions 1 assignee Claimed by @zaleslaw View on GitHub
bug files
Dominant language
Kotlin
Stars
1.1k
Forks
83
Avg merge
4d 12h
Merged PRs (30d)
30

Description

One value that cannot be converted to the target Arrow type fails the entire write — no file is produced —
when it sits in a column inside a `ColumnGroup`. `Mode.LOYAL` is supposed to degrade such a value and report
it to `mismatchSubscriber` instead of aborting, and it does exactly that for a top-level column.

The reason is where the conversion happens: a `ColumnGroup` written into an `ArrowType.Struct` field has its
columns converted inside `ArrowWriterImpl.infillVector`, outside the `try/catch` in `allocateVectorAndInfill`
that applies `ArrowWriter.Mode` to a failed conversion. So the exception goes straight out of the writer and
the subscriber is never called.

```kotlin
val df = dataFrameOf(DataColumn.createColumnGroup("g", dataFrameOf("x")("abc", "def")))
val schema = Schema(listOf(Field("g", FieldType.nullable(ArrowType.Struct()),
listOf(Field("x", FieldType.nullable(ArrowType.Int(32, true)), null)))))
df.arrowWriter(schema, ArrowWriter.Mode.LOYAL) { println(it) }
.use { it.saveArrowFeatherToByteArray() }
// CellConversionException: Failed to convert 'abc' from kotlin.String to kotlin.Int in column 'x', row 0
// mismatchSubscriber: never called
```

Pre-existing and not specific to a vector type — the same happens for any child vector, e.g.
`is IntVector -> column.convertToInt()`. Noticed in #2063
(https://github.com/Kotlin/dataframe/pull/2063#discussion_r3981437295).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.