JDBC: Allow reading schemas without querying actual data
- Dominant language
- Kotlin
- Stars
- 1.1k
- Forks
- 83
- Avg merge
- 4d 12h
- Merged PRs (30d)
- 30
Description
When extracting the DataSchema from an SQL query/database (`DataFrameSchema.readSqlTable()`) we currently call `DataFrame.readSqlTable(..., limit = 1).schema()`. This can cause a couple of issues:
- Error when the query or SQL table does not return any rows (actually, maybe this is fine)
- Error when we lack permission to query data; we're only allowed to look at the schema
- Dependent on how/where the data is stored, it may be unexpected and expensive to read a row
There are multiple ways to read just the Schema without reading actual data:
- use `"SELECT * FROM $fullTableName WHERE 1=0"` to read 0 rows: for [example](https://github.com/Kotlin/dataframe/blob/eb32d0b427412de9cf6a2ea6d0d5cf2821d3d73a/dataframe-jdbc/src/main/kotlin/org/jetbrains/kotlinx/dataframe/io/readDataFrameSchema.kt#L475)
- Downside is that we still need to query the table, albeit with no rows
- use `Connection.metaData.getColumns()` to get all column metadata: for example: [1](https://github.com/Kotlin/dataframe/blob/eb32d0b427412de9cf6a2ea6d0d5cf2821d3d73a/dataframe-jdbc/src/main/kotlin/org/jetbrains/kotlinx/dataframe/io/readDataFrameSchema.kt#L108) and [2](https://github.com/Kotlin/dataframe/blob/eb32d0b427412de9cf6a2ea6d0d5cf2821d3d73a/dataframe-jdbc/src/main/kotlin/org/jetbrains/kotlinx/dataframe/io/db/DbType.kt#L280).
- This is the most complete solution, but this usually doesn't give us the fully qualified java type for columns, requiring a watertight TYPE_NAME mapping on the implementor side
Contributor guide
Research direction
Start at DataFrameSchema.readSqlTable() in data/dataframe-jdbc/src/main/kotlin/org/jetbrains/kotlinx/dataframe/io/readDataFrameSchema.kt and trace its current DataFrame.readSqlTable(..., limit = 1). Compare the zero-row query example there with Connection.metaData.getColumns() and DbType.kt; done means schema extraction avoids reading actual data while preserving column types and existing behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kotlin, sql
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100