Kotlin / Kotlin/dataframe

Parallel computations

Open
#723 0 comments 0 reactions 0 assignees View on GitHub
enhancement performance research
Dominant language
Kotlin
Stars
1.1k
Forks
83
Avg merge
4d 12h
Merged PRs (30d)
30

Description

Object properties sometimes can be heavy or lazily computed and overall conversion take minutes for somewhat big lists
One can write this fairly simple code to speed up the conversion
```
val df = runBlocking {
list
.chunked(workload)
.map {
async(Dispatchers.IO) { it.toDataFrame() }
}.awaitAll().concat()
}
```
Although code is simple, it seems hard to properly make this parallelism part of the `toDataFrame` implementation.
Only `list.toDataFrame(maxDepth = int)` and `list.toDataFrame { properties(maxDepth = int) { } }` are side effect free, and it's (mostly) safe to split the list in chunks, run conversion in parallel and concat results. But even computation of the properties can be not parallel friendly. And then there is a question how workload is split and so on.

`add` and `convert` can be heavy and involve IO too. For this i have something like this in mind
```
fun DataFrame<*>.awaitAll(selector: ColumnSelector<*, Deferred<*>>) = runBlocking {
val column = getColumn(selector)
val values = column.toList().awaitAll()
replace(selector).with(values.toColumn(column.name(), infer = Infer.Type))
}
```

Usage:
```
val df = runBlocking {
otherDf.add("col") {
async(Dispatchers.IO) {
heavyCompute()
}
}.awaitAll { "col"() }
}
```

These two approaches can speed up dataframe code significantly in certain scenarios, so we can give them some visibility in the documentation.

Contributor guide

Open the contributing guide

Research direction

No implementation files or tests are named. Start by reviewing the proposed chunked toDataFrame approach and the DataFrame.awaitAll API in the issue, then clarify the intended scope and API design; done requires an agreed parallelism approach and corresponding documentation.

Written by the indexing model from the issue text.

Assessment

Tech stack
kotlin
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.