Avoid duplicate column type definitions by using DataSchema during CSV parsing
- Dominant language
- Kotlin
- Stars
- 1.1k
- Forks
- 83
- Avg merge
- 4d 12h
- Merged PRs (30d)
- 30
Description
### Problem
When reading CSV into a known `@DataSchema`, users must define column types twice:
- in the schema
- in `readCsv(colTypes = ...)`
Example:
```kt
@DataSchema
data class BusinessEmployee(
@ColumnName("ID")
val id: String
)
DataFrame.readCsv(
inputStream = ...,
colTypes = mapOf("ID" to ColType.String)
).cast()
```
This duplication is error-prone.
If `colTypes` is omitted, type inference may parse "0123" as a number, losing leading zeros.
### Context
CSV is a primary entry point into DataFrame workflows.
Parsing errors at this stage can silently corrupt data.
### Expected
Allow schema to participate in parsing so column types are defined once.
### Design decision
Choose one approach:
- schema-aware readCsv()
- passing schema into readCsv
- or explicitly keep current behavior and document limitation
### Acceptance criteria
- Decision made on API shape
- If implemented:
- schema types are applied during parsing
- colTypes overrides schema when specified
- test covers leading-zero case ("0123")
- If not implemented:
- limitation documented with recommended pattern
### Motivation
- Eliminates duplication between schema and parser configuration
- Prevents silent data corruption (e.g. losing leading zeros)
- Improves correctness and usability of CSV ingestion before 1.0
### Discussed in https://github.com/Kotlin/dataframe/discussions/1751
Originally posted by **sami-eljabali** March 16, 2026
Hi team
Thanks for the library! Big fan 😃
We're facing a pain point in reading CSV's, with its need to explicitly define column data types more than once. Otherwise risk them being parsed incorrectly.
Take for instance the following:
```kotlin
@DataSchema
data class BusinessEmployee(
@ColumnName("ID")
val id: String,
@ColumnName("FIRST_NAME")
val firstName: String,
@ColumnName("LAST_NAME")
val lastName: String,
}
```
```kotlin
DataFrame
.readCsv(
inputStream = ByteArrayInputStream(data),
colTypes = mapOf("ID" to ColType.String)
)
.cast()
.toList()
```
Notice how we twice define `ID` as type String.
We came across the need for this as when CSVs contains ID values appearing to be integers, ie `0123`, when parsed, are parsed as integers thereby omitted leading `0`'s.
Is there a way to avoid redefining the colTypes? Would like columns and their types be defined once in the DataSchema file. This would avoid duplication of code, and the possibility of devs forgetting to add secondary definitions.
Thanks again!
Contributor guide
Assessment
This issue has not been assessed yet.