apache / apache/arrow-rs

Add a way to map RecordBatch schema from one to another

Open
#5,996 4 comments 2 reactions 0 assignees View on GitHub
arrow enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 14h
Merged PRs (30d)
167

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
"Schema Evolution" is common when working with data -- this typically means that a table's overall schema may have columns that do not appear in individual `RecordBatch`

For example, maybe initially data was written with (`user`, `address`) and then later on a new column is added (`user`, `address`, `user_comment`).

All previously written data (e.g. in parquet files) might only have (`user`, `address`)

| user | address |
|--------|--------|
| Andrew | USA |
| Marco | Germany |

but new data has all three columns (`user`, `address`, `user_comment`).

| user | address | user_comment |
|--------|--------|--------|
| Andy | USA | Creator of DataFusion |

To query this data as a single table typically requires the data to have the same schema, and what query engines like Apache DataFusion or Apache Iceberg typically do is expand the old batches with null values, resulting in

| user | address | user_comment |
|--------|--------|--------|
| Andrew | USA | `NULL` |
| Marco | Germany | `NULL` |
| Andy | USA | Creator of DataFusion |

**Describe the solution you'd like**
I would like
1. an API in arrow-rs to do this mapping
2. Documentation and examples showing how to use it

**Describe alternatives you've considered**

#### SchemaMapper
One alternative would be to bring the `SchemaMapper` from DataFusion upstream to arrow-rs. This implementation has the benefit that it calculates the mapping once and then applies it to each batch

https://github.com/apache/datafusion/blob/b76c1b70506e45880f8e4bc5f9dc2ef088053acb/datafusion/core/src/datasource/schema_adapter.rs#L105-L174

#### `RecordBatch::ensure_schema`

Another alternative would be to add a `RecordBatch::ensure_schema` or something modeled on:

https://github.com/influxdata/influxdb3_core/blob/0f5ecbd6b17f83f7ad4ba55699fc2cd3e151cf94/arrow_util/src/util.rs#L28-L47

where the API is smaller but it is less efficient

#### Leave / make public in DataFusion (rather than arrow-rs)

Another alternative might be to leave this API in DataFurstion (and make it public) rather than upstreaming it to arrow-rs

I am proposing adding it to arrow-rs as I think it is generally useful (we use it in InfluxDB several places, for example)

**Additional context**

Contributor guide

Open the contributing guide

Research direction

Start by comparing DataFusion's schema_adapter.rs implementation with the linked InfluxDB arrow_util example, then inspect the arrow-rs RecordBatch and Schema APIs. Define the mapping behavior and API, add documentation and examples, and cover schema evolution cases with tests showing missing fields become null.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.