Parallel computations
- Dominant language
- Kotlin
- Stars
- 1.1k
- Forks
- 83
- Avg merge
- 4d 12h
- Merged PRs (30d)
- 30
Description
Object properties sometimes can be heavy or lazily computed and overall conversion take minutes for somewhat big lists
One can write this fairly simple code to speed up the conversion
```
val df = runBlocking {
list
.chunked(workload)
.map {
async(Dispatchers.IO) { it.toDataFrame() }
}.awaitAll().concat()
}
```
Although code is simple, it seems hard to properly make this parallelism part of the `toDataFrame` implementation.
Only `list.toDataFrame(maxDepth = int)` and `list.toDataFrame { properties(maxDepth = int) { } }` are side effect free, and it's (mostly) safe to split the list in chunks, run conversion in parallel and concat results. But even computation of the properties can be not parallel friendly. And then there is a question how workload is split and so on.
`add` and `convert` can be heavy and involve IO too. For this i have something like this in mind
```
fun DataFrame<*>.awaitAll(selector: ColumnSelector<*, Deferred<*>>) = runBlocking {
val column = getColumn(selector)
val values = column.toList().awaitAll()
replace(selector).with(values.toColumn(column.name(), infer = Infer.Type))
}
```
Usage:
```
val df = runBlocking {
otherDf.add("col") {
async(Dispatchers.IO) {
heavyCompute()
}
}.awaitAll { "col"() }
}
```
These two approaches can speed up dataframe code significantly in certain scenarios, so we can give them some visibility in the documentation.
Contributor guide
Research direction
No implementation files or tests are named. Start by reviewing the proposed chunked toDataFrame approach and the DataFrame.awaitAll API in the issue, then clarify the intended scope and API design; done requires an agreed parallelism approach and corresponding documentation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kotlin
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100