Should `DFSchema::datatype_is_logically_equal` be extended with more logical equality?
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
Apparently this function considers utf8 & utf8view types to be logically equal:
https://github.com/apache/datafusion/blob/f27e50c13647efa5292d5ef5b1e4ed6af3015962/datafusion/common/src/dfschema.rs#L668-L675
https://github.com/apache/datafusion/blob/f27e50c13647efa5292d5ef5b1e4ed6af3015962/datafusion/common/src/dfschema.rs#L726-L729
This raises the question of it we should extend it for more logical equality. For example:
- binary & binaryview should be logically equivalent
- should largeutf8, utf8, and utf8view all be logically equivalent? (and also for binary, including fixedsizebinary)
- what about list, largelist, listview, largelistview?
Origin PR:
- https://github.com/apache/datafusion/pull/15634
Contributor guide
Research direction
Start in datafusion/common/src/dfschema.rs at DFSchema::datatype_is_logically_equal and inspect the existing utf8/utf8view handling, then read the linked origin PR for context. Compare the proposed binary, large string, fixed-size binary, list, and view-type cases; done means the intended equivalence rules are decided and consistently covered in the relevant tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100