Kotlin / Kotlin/dataframe

JoinWith operations that can introduce nulls (leftJoin, rightJoin, etc) produce FrameColumns with misleading schema resulting in potential column not found exceptions

Open
#2,040 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Kotlin
Stars
1.1k
Forks
83
Avg merge
4d 12h
Merged PRs (30d)
30

Description

Repro:
```kt
val l = (0..10).toDataFrame {
"id" from { it }
}
val r = (0..5).toDataFrame {
"id" from { it }
"frame" from {
(0..10).toDataFrame {
"col" from { it }
}
}
}
val res = l.leftJoinWith(r) { id == right.id }
res.schema().print()
res.frame[6].col
```

Actual:
```
id: Int
id1: Int?
frame: *
col: Int

Exception in thread "main" java.lang.IllegalStateException: Column not found exception in the generated DataFrame extension property 'col': Column not found: 'col'. See https://kotlin.github.io/dataframe/data-schemas-and-extension-properties-troubleshooting.html for more information.
at org.jetbrains.kotlinx.dataframe.exceptions.ExtensionPropertyExceptionKt.handleExtensionPropertyException(extensionPropertyException.kt:32)
at JoinFrameKt$main$res$1$DataFramePropertiesScope1.getCol(joinFrame.kt)
at JoinFrameKt.main(joinFrame.kt:18)
at JoinFrameKt.main(joinFrame.kt)
```

Both runtime and compile schemas tell us there's col in frame, but it's not always the case, and simply iterating over `res` can lead to an unexpected column not found exception.

Image

Root cause:
Potential fix touches broader issue. `joinWithImpl` uses createByInference, which produces this invalid result.
https://github.com/Kotlin/dataframe/blob/8d7d9bd6af7e27f2c06c17cf9f6c72af68d36b11/core/src/main/kotlin/org/jetbrains/kotlinx/dataframe/impl/api/joinWith.kt#L114

https://github.com/Kotlin/dataframe/blob/0d009c68649864f64b3ae3f231235bc951a63788/core/src/main/kotlin/org/jetbrains/kotlinx/dataframe/impl/columns/constructors.kt#L281

Smaller repro that would involve only `createByInference` will be

```kt
fun main() {
val res = dataFrameOf("frame" to columnOf(null, dataFrameOf("col" to columnOf(1))))
res.cast().frame[0].col
}

@DataSchema
data class DataFrameOf_42(
val frame: List
) {
@DataSchema
data class Frame(
val col: Int
)
}
```

Contributor guide

Open the contributing guide

Research direction

Start with core/src/main/kotlin/org/jetbrains/kotlinx/dataframe/impl/api/joinWith.kt at joinWithImpl and trace its use of createByInference. Then inspect core/src/main/kotlin/org/jetbrains/kotlinx/dataframe/impl/columns/constructors.kt and reproduce the smaller null-containing frame case. Done means leftJoinWith and createByInference produce schemas whose nested FrameColumns remain usable when nulls are present, without the reported 'col' lookup exception.

Written by the indexing model from the issue text.

Assessment

Tech stack
kotlin
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.