Row-based DataFrame builder/collector DSL
- Dominant language
- Kotlin
- Stars
- 1.1k
- Forks
- 83
- Avg merge
- 4d 12h
- Merged PRs (30d)
- 30
Description
We could consider adding a DSL similar to `buildList {}` but `buildDataFrame {}` where rows can be `yielded` one at a time.
This will be more performant than:
```kt
var df = DataFrame.EMPTY
for (row in data) {
df = df.append(row....)
}
```
It could use something like our [ColumnDataCollector](https://github.com/Kotlin/dataframe/blob/master/core/src/main/kotlin/org/jetbrains/kotlinx/dataframe/impl/ColumnDataCollector.kt) under-the-hood but one data collector for each column the user wants to be in the dataframe.
Ideas for the DSL:
1) append-like
```kt
val df = buildDataFrame("a", "b", "c") {
for (row in data) {
if (row.isSomething) continue
append(row.myA, row.myB, row.myC)
}
}
```
or
```kt
val df = buildDataFrame {
for (row in data) {
if (row.isSomething) continue
append("a" to row.myA, "b" to row.myB, "c" to row.myC)
}
}
```
2) map-like
```kt
val df = buildDataFrame {
for (row in data) {
if (row.isSomething) continue
add("a", row.myA) // or put?
this["b"] = row.myB
put("c", row.myC) // or add?
}
}
```
3) `toDataFrame`/`add`-like
```kt
val df = buildDataFrame {
for (row in data) {
if (row.isSomething) continue
"a" from { row.myA }
"b" from row.myB // {} are not needed here because we have a single row
row.myC into "c"
}
}
```
(This should not be confused with the column-based [DynamicDataFrameBuilder](https://github.com/Kotlin/dataframe/blob/7c189f0f5788de1dc28515c4bdd4d0c4928b2f22/core/src/main/kotlin/org/jetbrains/kotlinx/dataframe/api/constructors.kt#L488))
Obligatory mention: We can already do something similar like this is the latest `-dev` version, though it creates `n` DataFrames under the hood:
```kt
val df = buildList {
for (row in data) {
if (row.isSomething) continue
this += mapOf("a" to row.myA, "b" to row.myB, "c" to row.myC).toDataRow()
}
}.concat()
```
Contributor guide
Research direction
Start by reading core/src/main/kotlin/org/jetbrains/kotlinx/dataframe/impl/ColumnDataCollector.kt and the DynamicDataFrameBuilder in core/src/main/kotlin/org/jetbrains/kotlinx/dataframe/api/constructors.kt. Compare those APIs with the existing buildList, toDataRow, and concat approach shown in the issue. Done requires an agreed row-based DSL design and an implementation that collects rows without creating one DataFrame per row.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kotlin
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100