Kotlin / Kotlin/dataframe

Define column type specification API for data readers

Open
#340 4 comments 0 reactions 1 assignee Claimed by @zaleslaw View on GitHub
API enhancement research
Dominant language
Kotlin
Stars
1.1k
Forks
83
Avg merge
4d 12h
Merged PRs (30d)
30

Description

### Problem

Column type guessing often does not match user expectations when reading data.

Users sometimes need to explicitly specify how columns should be parsed, especially for formats like Excel or CSV where automatic inference can produce wrong types.

Similar APIs exist in other tabular libraries, for example `col_types` in readr or compact column specs in krangl.

### Current gap

DataFrame currently does not have a unified column specification API across readers.

As a result:
- users must manually fix inferred types after reading
- parsing behavior differs between formats
- adding support format-by-format risks inconsistent APIs

### Design question

What should the column specification API look like?

Options to evaluate:
- map-based configuration, e.g. `colTypes = mapOf("id" to ColType.String)`
- compact column spec syntax, e.g. `"sis??bbsds"`
- shared `ParserOptions` / reader configuration object
- reader-specific configuration where necessary

### Expected

Define a consistent API for specifying column types when reading data.

### Acceptance criteria

- Decide supported column specification model
- Define how it applies to at least CSV and Excel
- Ensure default type inference remains unchanged
- Add tests for explicit column type specification
- Document usage with examples

### Motivation

Data reading is one of the first steps in most DataFrame workflows.

Incorrect type inference can lead to wrong data types and require manual cleanup after loading. A clear column specification API improves reliability and makes readers more predictable before 1.0.

Very often column type guessing does not work as intended by the user. Thus it would be helpful to support an optional column spec. This is very common in other tabular apis, see e.g. `col_types` argument in https://readr.tidyverse.org/reference/read_delim.html#arguments

Note: krangl supported this for all readers previously in a very similar manner:
```
DataFrame.readExcel(
excelFile.absolutePath,
sheet = "Equipment",
colTypes = CompactColumnSpec("sis??bbsds")
)
```
See https://github.com/holgerbrandl/krangl/blob/master/src/main/kotlin/krangl/TableIO.kt#L226

It would be great if kdf could be extended similarly to support a wider set of input formats directly without manual correction (as suggested in https://kotlin.github.io/dataframe/read.html#reading-excel).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.