apache / apache/datafusion

Improve schema merging

Open
#4,223 2 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

I am trying to work with the nyctaxi parquet data set which has one file per month. Over time, some of the types have changed. For example `passenger_count` started out as `Int64` and was later changed to `Float64`.

Arrow-rs can not merge these schemas.

Other solutions (such as DuckDB) will merge these schemas and pick the least restrictive type (`Float64`).

**Describe the solution you'd like**

**Describe alternatives you've considered**

**Additional context**

Contributor guide

Open the contributing guide

Research direction

No source files or tests are named. Start by reproducing the failure with the monthly nyctaxi Parquet files and inspect the Arrow-rs schema-merging entry point; done means schemas containing Int64 and Float64 passenger_count merge successfully using Float64.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.