apache / apache/arrow-rs

Support for "Schema evolution" / Schema Adapters

Open
#6,735 8 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

Sometimes it is desired to convert RecordBatches from one schema so they match another. This is often done when data is stored in several different sources (like parquet files) that are "compatible" but not exactly the same (e.g. maybe newer files have new columns)

Common transformations are:
- Reorder columns by name (so a file with `(a int, b char)` and one with `(b char, a int)` could be read as a single stream
- Insert missing columns (so a file with `(a int, b char)` and a file with `(a int)` could be merged

It is also common to want to fill in missing columns with either `Null` or some constant (e.g. `0`) so a controllable policy would be nice

Note that these these usecases are pretty similar to casting Structs (e.g. reordering fields with the same name but different position)
- https://github.com/apache/arrow-rs/issues/4908

Often computing the transformation may be non trivial (e.g. matching columns by name) so it would be nice to do the mapping calculation once per schema rather than once per batch / StructArrayschema. For example DF's SchemaAdapter computes the mapping once and can then apply that to multiple batches.

**Describe the solution you'd like**
Add some API in Arrow-rs to do this mapping

One alternative, suggested by @tustvold would be to add a first-party schema adapter into arrow-rs.

**Describe alternatives you've considered**

For anyone interested, here is the API that is in DataFusion (it now even has ASCII art and Examples, thanks to @itsjunetime and myself):

- https://docs.rs/datafusion/latest/datafusion/datasource/schema_adapter/struct.DefaultSchemaAdapterFactory.html
Screenshot 2024-11-13 at 6 57 28 AM

We can/should probably change the names and reduce the levels of indirection of we upstreamed this into arrow-rs
**Additional context**

Contributor guide

Open the contributing guide

Research direction

Start with the linked DataFusion DefaultSchemaAdapterFactory API and the RecordBatch and StructArray use cases described in the issue. Clarify the upstream API, naming, and policies for reordering, missing columns, and Null or constant fills; done requires an agreed schema-mapping design and coverage for these transformations.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.