JoinWith operations that can introduce nulls (leftJoin, rightJoin, etc) produce FrameColumns with misleading schema resulting in potential column not found exceptions
- Dominant language
- Kotlin
- Stars
- 1.1k
- Forks
- 83
- Avg merge
- 4d 12h
- Merged PRs (30d)
- 30
Description
Repro:
```kt
val l = (0..10).toDataFrame {
"id" from { it }
}
val r = (0..5).toDataFrame {
"id" from { it }
"frame" from {
(0..10).toDataFrame {
"col" from { it }
}
}
}
val res = l.leftJoinWith(r) { id == right.id }
res.schema().print()
res.frame[6].col
```
Actual:
```
id: Int
id1: Int?
frame: *
col: Int
Exception in thread "main" java.lang.IllegalStateException: Column not found exception in the generated DataFrame extension property 'col': Column not found: 'col'. See https://kotlin.github.io/dataframe/data-schemas-and-extension-properties-troubleshooting.html for more information.
at org.jetbrains.kotlinx.dataframe.exceptions.ExtensionPropertyExceptionKt.handleExtensionPropertyException(extensionPropertyException.kt:32)
at JoinFrameKt$main$res$1$DataFramePropertiesScope1.getCol(joinFrame.kt)
at JoinFrameKt.main(joinFrame.kt:18)
at JoinFrameKt.main(joinFrame.kt)
```
Both runtime and compile schemas tell us there's col in frame, but it's not always the case, and simply iterating over `res` can lead to an unexpected column not found exception.
Root cause:
Potential fix touches broader issue. `joinWithImpl` uses createByInference, which produces this invalid result.
https://github.com/Kotlin/dataframe/blob/8d7d9bd6af7e27f2c06c17cf9f6c72af68d36b11/core/src/main/kotlin/org/jetbrains/kotlinx/dataframe/impl/api/joinWith.kt#L114
https://github.com/Kotlin/dataframe/blob/0d009c68649864f64b3ae3f231235bc951a63788/core/src/main/kotlin/org/jetbrains/kotlinx/dataframe/impl/columns/constructors.kt#L281
Smaller repro that would involve only `createByInference` will be
```kt
fun main() {
val res = dataFrameOf("frame" to columnOf(null, dataFrameOf("col" to columnOf(1))))
res.cast().frame[0].col
}
@DataSchema
data class DataFrameOf_42(
val frame: List
) {
@DataSchema
data class Frame(
val col: Int
)
}
```
Contributor guide
Research direction
Start with core/src/main/kotlin/org/jetbrains/kotlinx/dataframe/impl/api/joinWith.kt at joinWithImpl and trace its use of createByInference. Then inspect core/src/main/kotlin/org/jetbrains/kotlinx/dataframe/impl/columns/constructors.kt and reproduce the smaller null-containing frame case. Done means leftJoinWith and createByInference produce schemas whose nested FrameColumns remain usable when nulls are present, without the reported 'col' lookup exception.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kotlin
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100