Add a way to map RecordBatch schema from one to another
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 167
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
"Schema Evolution" is common when working with data -- this typically means that a table's overall schema may have columns that do not appear in individual `RecordBatch`
For example, maybe initially data was written with (`user`, `address`) and then later on a new column is added (`user`, `address`, `user_comment`).
All previously written data (e.g. in parquet files) might only have (`user`, `address`)
| user | address |
|--------|--------|
| Andrew | USA |
| Marco | Germany |
but new data has all three columns (`user`, `address`, `user_comment`).
| user | address | user_comment |
|--------|--------|--------|
| Andy | USA | Creator of DataFusion |
To query this data as a single table typically requires the data to have the same schema, and what query engines like Apache DataFusion or Apache Iceberg typically do is expand the old batches with null values, resulting in
| user | address | user_comment |
|--------|--------|--------|
| Andrew | USA | `NULL` |
| Marco | Germany | `NULL` |
| Andy | USA | Creator of DataFusion |
**Describe the solution you'd like**
I would like
1. an API in arrow-rs to do this mapping
2. Documentation and examples showing how to use it
**Describe alternatives you've considered**
#### SchemaMapper
One alternative would be to bring the `SchemaMapper` from DataFusion upstream to arrow-rs. This implementation has the benefit that it calculates the mapping once and then applies it to each batch
https://github.com/apache/datafusion/blob/b76c1b70506e45880f8e4bc5f9dc2ef088053acb/datafusion/core/src/datasource/schema_adapter.rs#L105-L174
#### `RecordBatch::ensure_schema`
Another alternative would be to add a `RecordBatch::ensure_schema` or something modeled on:
https://github.com/influxdata/influxdb3_core/blob/0f5ecbd6b17f83f7ad4ba55699fc2cd3e151cf94/arrow_util/src/util.rs#L28-L47
where the API is smaller but it is less efficient
#### Leave / make public in DataFusion (rather than arrow-rs)
Another alternative might be to leave this API in DataFurstion (and make it public) rather than upstreaming it to arrow-rs
I am proposing adding it to arrow-rs as I think it is generally useful (we use it in InfluxDB several places, for example)
**Additional context**
Contributor guide
Research direction
Start by comparing DataFusion's schema_adapter.rs implementation with the linked InfluxDB arrow_util example, then inspect the arrow-rs RecordBatch and Schema APIs. Define the mapping behavior and API, add documentation and examples, and cover schema evolution cases with tests showing missing fields become null.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100