Kotlin / Kotlin/dataframe

Row-based DataFrame builder/collector DSL

Open
#1,541 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Kotlin
Stars
1.1k
Forks
83
Avg merge
4d 12h
Merged PRs (30d)
30

Description

We could consider adding a DSL similar to `buildList {}` but `buildDataFrame {}` where rows can be `yielded` one at a time.

This will be more performant than:

```kt
var df = DataFrame.EMPTY
for (row in data) {
df = df.append(row....)
}
```

It could use something like our [ColumnDataCollector](https://github.com/Kotlin/dataframe/blob/master/core/src/main/kotlin/org/jetbrains/kotlinx/dataframe/impl/ColumnDataCollector.kt) under-the-hood but one data collector for each column the user wants to be in the dataframe.

Ideas for the DSL:

1) append-like
```kt
val df = buildDataFrame("a", "b", "c") {
for (row in data) {
if (row.isSomething) continue

append(row.myA, row.myB, row.myC)
}
}
```
or
```kt
val df = buildDataFrame {
for (row in data) {
if (row.isSomething) continue

append("a" to row.myA, "b" to row.myB, "c" to row.myC)
}
}
```

2) map-like
```kt
val df = buildDataFrame {
for (row in data) {
if (row.isSomething) continue

add("a", row.myA) // or put?
this["b"] = row.myB
put("c", row.myC) // or add?
}
}
```

3) `toDataFrame`/`add`-like
```kt
val df = buildDataFrame {
for (row in data) {
if (row.isSomething) continue

"a" from { row.myA }
"b" from row.myB // {} are not needed here because we have a single row
row.myC into "c"
}
}
```

(This should not be confused with the column-based [DynamicDataFrameBuilder](https://github.com/Kotlin/dataframe/blob/7c189f0f5788de1dc28515c4bdd4d0c4928b2f22/core/src/main/kotlin/org/jetbrains/kotlinx/dataframe/api/constructors.kt#L488))

Obligatory mention: We can already do something similar like this is the latest `-dev` version, though it creates `n` DataFrames under the hood:
```kt
val df = buildList {
for (row in data) {
if (row.isSomething) continue
this += mapOf("a" to row.myA, "b" to row.myB, "c" to row.myC).toDataRow()
}
}.concat()
```

Contributor guide

Open the contributing guide

Research direction

Start by reading core/src/main/kotlin/org/jetbrains/kotlinx/dataframe/impl/ColumnDataCollector.kt and the DynamicDataFrameBuilder in core/src/main/kotlin/org/jetbrains/kotlinx/dataframe/api/constructors.kt. Compare those APIs with the existing buildList, toDataRow, and concat approach shown in the issue. Done requires an agreed row-based DSL design and an implementation that collects rows without creating one DataFrame per row.

Written by the indexing model from the issue text.

Assessment

Tech stack
kotlin
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.