Define column type specification API for data readers
- Dominant language
- Kotlin
- Stars
- 1.1k
- Forks
- 83
- Avg merge
- 4d 12h
- Merged PRs (30d)
- 30
Description
### Problem
Column type guessing often does not match user expectations when reading data.
Users sometimes need to explicitly specify how columns should be parsed, especially for formats like Excel or CSV where automatic inference can produce wrong types.
Similar APIs exist in other tabular libraries, for example `col_types` in readr or compact column specs in krangl.
### Current gap
DataFrame currently does not have a unified column specification API across readers.
As a result:
- users must manually fix inferred types after reading
- parsing behavior differs between formats
- adding support format-by-format risks inconsistent APIs
### Design question
What should the column specification API look like?
Options to evaluate:
- map-based configuration, e.g. `colTypes = mapOf("id" to ColType.String)`
- compact column spec syntax, e.g. `"sis??bbsds"`
- shared `ParserOptions` / reader configuration object
- reader-specific configuration where necessary
### Expected
Define a consistent API for specifying column types when reading data.
### Acceptance criteria
- Decide supported column specification model
- Define how it applies to at least CSV and Excel
- Ensure default type inference remains unchanged
- Add tests for explicit column type specification
- Document usage with examples
### Motivation
Data reading is one of the first steps in most DataFrame workflows.
Incorrect type inference can lead to wrong data types and require manual cleanup after loading. A clear column specification API improves reliability and makes readers more predictable before 1.0.
Very often column type guessing does not work as intended by the user. Thus it would be helpful to support an optional column spec. This is very common in other tabular apis, see e.g. `col_types` argument in https://readr.tidyverse.org/reference/read_delim.html#arguments
Note: krangl supported this for all readers previously in a very similar manner:
```
DataFrame.readExcel(
excelFile.absolutePath,
sheet = "Equipment",
colTypes = CompactColumnSpec("sis??bbsds")
)
```
See https://github.com/holgerbrandl/krangl/blob/master/src/main/kotlin/krangl/TableIO.kt#L226
It would be great if kdf could be extended similarly to support a wider set of input formats directly without manual correction (as suggested in https://kotlin.github.io/dataframe/read.html#reading-excel).
Contributor guide
Assessment
This issue has not been assessed yet.