Kotlin / Kotlin/dataframe

Add Redshift support as an example with Custom Database

Open
#549 2 comments 0 reactions 1 assignee Claimed by @zaleslaw View on GitHub
databases examples
Dominant language
Kotlin
Stars
1.1k
Forks
83
Avg merge
4d 12h
Merged PRs (30d)
30

Description

Attempted to try out the new SQL support.

I used the Amazon provided [JDBC driver](https://docs.aws.amazon.com/redshift/latest/mgmt/jdbc20-download-driver.html) and the connection oriented APIs. Redshift is a flavor of postgres and the driver should handle the differences/mappings so I figured it may just work.

Example Kotlin notebook

```kotlin
%use dataframe(0.12.0)
@file:DependsOn("../redshift-jdbc42-2.1.0.24.jar")

import com.amazon.redshift.jdbc.Driver
import com.amazon.redshift.jdbc.DataSource

data class DatabaseConfig(
val endpoint: String,
val database: String,
val user: String,
val password: String,
val port: Int
) {
val url: String = "jdbc:redshift://$endpoint:$port/$database"
}

val dbConfig = DatabaseConfig(...)

val ds = DataSource().apply {
setURL(dbConfig.url)
user = dbConfig.user
password = dbConfig.password
}

val query = "..."

val df = ds.getConnection().use { conn ->
DataFrame.readSqlQuery(conn, query)
}

```

Results in:
```
Only H2, MariaDB, MySQL, SQLite and PostgreSQL are supported!
java.lang.IllegalArgumentException: Unsupported database type in the url: jdbc:redshift://
Only H2, MariaDB, MySQL, SQLite and PostgreSQL are supported!
at org.jetbrains.kotlinx.dataframe.io.db.UtilKt.extractDBTypeFromUrl(util.kt:20)
at org.jetbrains.kotlinx.dataframe.io.ReadJdbcKt.readSqlQuery(readJdbc.kt:183)
at org.jetbrains.kotlinx.dataframe.io.ReadJdbcKt.readSqlQuery(readJdbc.kt:168)
at Line_22_jupyter.(Line_22.jupyter.kts:12)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480)
at kotlin.script.experimental.jvm.BasicJvmScriptEvaluator.evalWithConfigAndOtherScriptsResults(BasicJvmScriptEvaluator.kt:105)
at kotlin.script.experimental.jvm.BasicJvmScriptEvaluator.invoke$suspendImpl(BasicJvmScriptEvaluator.kt:47)
at kotlin.script.experimental.jvm.BasicJvmScriptEvaluator.invoke(BasicJvmScriptEvaluator.kt)
at kotlin.script.experimental.jvm.BasicJvmReplEvaluator.eval(BasicJvmReplEvaluator.kt:49)
at org.jetbrains.kotlinx.jupyter.repl.impl.InternalEvaluatorImpl$eval$resultWithDiagnostics$1.invokeSuspend(InternalEvaluatorImpl.kt:107)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:284)
at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:85)
at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:59)
at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt:38)
at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source)
at org.jetbrains.kotlinx.jupyter.repl.impl.InternalEvaluatorImpl.eval(InternalEvaluatorImpl.kt:107)
at org.jetbrains.kotlinx.jupyter.repl.impl.CellExecutorImpl$execute$1$result$1.invoke(CellExecutorImpl.kt:78)
at org.jetbrains.kotlinx.jupyter.repl.impl.CellExecutorImpl$execute$1$result$1.invoke(CellExecutorImpl.kt:76)
at org.jetbrains.kotlinx.jupyter.ReplForJupyterImpl.withHost(repl.kt:681)
at org.jetbrains.kotlinx.jupyter.repl.impl.CellExecutorImpl.execute(CellExecutorImpl.kt:76)
at org.jetbrains.kotlinx.jupyter.repl.CellExecutor$DefaultImpls.execute$default(CellExecutor.kt:16)
at org.jetbrains.kotlinx.jupyter.ReplForJupyterImpl$evalEx$1.invoke(repl.kt:496)
at org.jetbrains.kotlinx.jupyter.ReplForJupyterImpl$evalEx$1.invoke(repl.kt:474)
at org.jetbrains.kotlinx.jupyter.ReplForJupyterImpl.withEvalContext(repl.kt:437)
at org.jetbrains.kotlinx.jupyter.ReplForJupyterImpl.evalEx(repl.kt:474)
at org.jetbrains.kotlinx.jupyter.messaging.ProtocolKt$shellMessagesHandler$2$res$1.invoke(protocol.kt:320)
at org.jetbrains.kotlinx.jupyter.messaging.ProtocolKt$shellMessagesHandler$2$res$1.invoke(protocol.kt:314)
at org.jetbrains.kotlinx.jupyter.JupyterExecutorImpl$runExecution$execThread$1.invoke(execution.kt:38)
at org.jetbrains.kotlinx.jupyter.JupyterExecutorImpl$runExecution$execThread$1.invoke(execution.kt:33)
at kotlin.concurrent.ThreadsKt$thread$thread$1.run(Thread.kt:30)
```

It's unclear to me how much work it would be to support Redshift or if using the custom provided driver if it should just work out of the box (using the connection oriented APIs of course). It seems like perhaps the connection oriented APIs though shouldn't be checking the database type or perhaps an escape hatch to disable it?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.