Kotlin / Kotlin/dataframe

Support DuckDB as computation backend

Open
#186 11 comments 0 reactions 1 assignee Claimed by @zaleslaw View on GitHub
databases enhancement research
Dominant language
Kotlin
Stars
1.1k
Forks
83
Avg merge
4d 12h
Merged PRs (30d)
30

Description

If I understand Kotlin dataframes correctly, computations are done directly in the JVM.
For many use-cases, the optimized data storage and vectorized computations of DuckDB could be very useful in terms of performance and memory consumption.

So maybe there could be a way to move computations to DuckDB via Apache Arrow, which offers a columnar memory format, zero-copy reads, streaming, and an extensive Java API.

The goal would be to perform analytical computations against DuckDB directly from Kotlin in a way similar to the one described here for Python and R:
https://arrow.apache.org/blog/2021/12/03/arrow-duckdb/

Looking forward to hear what you think about the feasibility of such an approach.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.