apache / apache/arrow-rs

Add a way to compare datatype for "semantic compatibility / equality"

Open
#3,199 8 comments 0 reactions 1 assignee Claimed by @Rich-T-kid View on GitHub
arrow enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
In https://github.com/apache/arrow-datafusion/issues/4346 from DataFusion, we have a check that between each optimization pass the schema hasn't changed (see https://github.com/apache/arrow-datafusion/pull/4233 by @jackwener).

However it turns out that some passes may change the metadata or nullability for a plan which is ok, but this causes the schema equality check to fail incorrectly

**Describe the solution you'd like**
I would like a function that does "semanic equality" checking of a datatype -- specifically one that ignores any metadata or nullability information and only compares:
1. field_name
2. data_type

**Describe alternatives you've considered**
We can keep the code entirely in datafusion -- see initial implementation in https://github.com/apache/arrow-datafusion/pull/4347

**Additional context**
As suggested by @mingmwang on https://github.com/apache/arrow-datafusion/pull/4347#issuecomment-1325885678

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.