Kotlin / Kotlin/dataframe

JDBC: Allow reading schemas without querying actual data

Open
#1,681 1 comment 0 reactions 0 assignees View on GitHub
databases enhancement
Dominant language
Kotlin
Stars
1.1k
Forks
83
Avg merge
4d 12h
Merged PRs (30d)
30

Description

When extracting the DataSchema from an SQL query/database (`DataFrameSchema.readSqlTable()`) we currently call `DataFrame.readSqlTable(..., limit = 1).schema()`. This can cause a couple of issues:
- Error when the query or SQL table does not return any rows (actually, maybe this is fine)
- Error when we lack permission to query data; we're only allowed to look at the schema
- Dependent on how/where the data is stored, it may be unexpected and expensive to read a row

There are multiple ways to read just the Schema without reading actual data:
- use `"SELECT * FROM $fullTableName WHERE 1=0"` to read 0 rows: for [example](https://github.com/Kotlin/dataframe/blob/eb32d0b427412de9cf6a2ea6d0d5cf2821d3d73a/dataframe-jdbc/src/main/kotlin/org/jetbrains/kotlinx/dataframe/io/readDataFrameSchema.kt#L475)
- Downside is that we still need to query the table, albeit with no rows
- use `Connection.metaData.getColumns()` to get all column metadata: for example: [1](https://github.com/Kotlin/dataframe/blob/eb32d0b427412de9cf6a2ea6d0d5cf2821d3d73a/dataframe-jdbc/src/main/kotlin/org/jetbrains/kotlinx/dataframe/io/readDataFrameSchema.kt#L108) and [2](https://github.com/Kotlin/dataframe/blob/eb32d0b427412de9cf6a2ea6d0d5cf2821d3d73a/dataframe-jdbc/src/main/kotlin/org/jetbrains/kotlinx/dataframe/io/db/DbType.kt#L280).
- This is the most complete solution, but this usually doesn't give us the fully qualified java type for columns, requiring a watertight TYPE_NAME mapping on the implementor side

Contributor guide

Open the contributing guide

Research direction

Start at DataFrameSchema.readSqlTable() in data/dataframe-jdbc/src/main/kotlin/org/jetbrains/kotlinx/dataframe/io/readDataFrameSchema.kt and trace its current DataFrame.readSqlTable(..., limit = 1). Compare the zero-row query example there with Connection.metaData.getColumns() and DbType.kt; done means schema extraction avoids reading actual data while preserving column types and existing behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
kotlin, sql
Domain
databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.