apache / apache/datafusion

Should `DFSchema::datatype_is_logically_equal` be extended with more logical equality?

Open
#24,137 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

Apparently this function considers utf8 & utf8view types to be logically equal:

https://github.com/apache/datafusion/blob/f27e50c13647efa5292d5ef5b1e4ed6af3015962/datafusion/common/src/dfschema.rs#L668-L675

https://github.com/apache/datafusion/blob/f27e50c13647efa5292d5ef5b1e4ed6af3015962/datafusion/common/src/dfschema.rs#L726-L729

This raises the question of it we should extend it for more logical equality. For example:

- binary & binaryview should be logically equivalent
- should largeutf8, utf8, and utf8view all be logically equivalent? (and also for binary, including fixedsizebinary)
- what about list, largelist, listview, largelistview?

Origin PR:

- https://github.com/apache/datafusion/pull/15634

Contributor guide

Open the contributing guide

Research direction

Start in datafusion/common/src/dfschema.rs at DFSchema::datatype_is_logically_equal and inspect the existing utf8/utf8view handling, then read the linked origin PR for context. Compare the proposed binary, large string, fixed-size binary, list, and view-type cases; done means the intended equivalence rules are decided and consistently covered in the relevant tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.