Add `fillWhenMissing` option to `convertTo` for schema-compatible CSV evolution
- Dominant language
- Kotlin
- Stars
- 1.1k
- Forks
- 83
- Avg merge
- 4d 12h
- Merged PRs (30d)
- 30
Description
### Problem
When using `convertTo()`, `fill {}` applies both to:
- rows where the column is missing
- rows where the column is present
This makes it impossible to handle evolving schemas correctly.
Typical case:
- schema is extended with a new column
- some CSV files contain the new column
- others do not
Current behavior:
- `fill` overwrites values even when the column exists
### Expected
Provide a way to apply default values **only when the column is missing**, without overriding existing data.
### Proposed solution
Add a `fillWhenMissing {}` option to `convertTo`.
Example:
`df.convertTo {
fillWhenMissing { col("newCol") }.with { "default" }
}`
### Acceptance criteria
- `fillWhenMissing` applies only when the column is absent in source data
- Existing values are preserved if the column exists
- Behavior is consistent with current `fill`
- Add test covering:
- missing column case
- present column case
### Motivation
Schema evolution is a common scenario when working with CSV and external data sources :contentReference[oaicite:0]{index=0}
Without this feature:
- users cannot safely introduce new fields
- existing data may be silently overwritten
This improves correctness and usability of `convertTo` in real-world pipelines before 1.0.
### Discussed in https://github.com/Kotlin/dataframe/discussions/1336
>
> Originally posted by **hantsy** July 22, 2025
> How to apply the `fill` to the condition that the fields is missing the csv but defind in the DataSchema.
>
>
> ```kotlin
> df.convertTo {
> fill { col("NewCol") }.with{ "default" }
> }
>
> ```
>
> Currently, I faced the situation like this, we updated the Data Schema and add some new columns.
>
> Some users use the new schema fields in the CSV file, and other not update and still used the old schema which lacked the new col here.
>
> The `fill` is good to apply the default to old format CSV if they do not provide the new col. But the fill will override the value in the new format CSV file.
>
> Is possible to only apply the `fill` to the old format here, and in the new format, use the value in CSV directly.
Adding a `fillWhenMissing {}` option would be a good solution for this
Contributor guide
Assessment
This issue has not been assessed yet.