Support for "Schema evolution" / Schema Adapters
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 169
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
Sometimes it is desired to convert RecordBatches from one schema so they match another. This is often done when data is stored in several different sources (like parquet files) that are "compatible" but not exactly the same (e.g. maybe newer files have new columns)
Common transformations are:
- Reorder columns by name (so a file with `(a int, b char)` and one with `(b char, a int)` could be read as a single stream
- Insert missing columns (so a file with `(a int, b char)` and a file with `(a int)` could be merged
It is also common to want to fill in missing columns with either `Null` or some constant (e.g. `0`) so a controllable policy would be nice
Note that these these usecases are pretty similar to casting Structs (e.g. reordering fields with the same name but different position)
- https://github.com/apache/arrow-rs/issues/4908
Often computing the transformation may be non trivial (e.g. matching columns by name) so it would be nice to do the mapping calculation once per schema rather than once per batch / StructArrayschema. For example DF's SchemaAdapter computes the mapping once and can then apply that to multiple batches.
**Describe the solution you'd like**
Add some API in Arrow-rs to do this mapping
One alternative, suggested by @tustvold would be to add a first-party schema adapter into arrow-rs.
**Describe alternatives you've considered**
For anyone interested, here is the API that is in DataFusion (it now even has ASCII art and Examples, thanks to @itsjunetime and myself):
- https://docs.rs/datafusion/latest/datafusion/datasource/schema_adapter/struct.DefaultSchemaAdapterFactory.html
We can/should probably change the names and reduce the levels of indirection of we upstreamed this into arrow-rs
**Additional context**
Contributor guide
Research direction
Start with the linked DataFusion DefaultSchemaAdapterFactory API and the RecordBatch and StructArray use cases described in the issue. Clarify the upstream API, naming, and policies for reordering, missing columns, and Null or constant fills; done requires an agreed schema-mapping design and coverage for these transformations.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100