[EPIC] Decouple logical from physical types
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
This epic tracks an ordered list tasks related to the proposal: Decouple logical from physical types (#11513). The goal is:
> Logical operators during logical planning should unquestionably **not** have access to the physical type information, which should exclusively be reserved to the physical planning and physical execution.
>
> LogicalPlans will use LogicalType while PhysicalPlans will use DataType.
### Describe the solution you'd like
Make `ScalarValue` values logical:
- [x] #12536 (merged in `logical-types`)
- [x] #12793 (merged in `logical-types`)
- [x] #12853
- [ ] Remove `ScalarValue::LargeUtf8 `and `ScalarValue::Utf8View` in favour of `ScalarValue::Utf8`
- [ ] Remove `ScalarValue::LargeBinary` and `ScalarValue::BinaryView` in favour of `ScalarValue::Binary`
- [ ] Remove `ScalarValue::Dictionary` (from https://github.com/apache/datafusion/pull/12488)
Contributor guide
Research direction
Review the proposal in #11513 and the remaining unchecked ScalarValue tasks, starting with the logical-types work referenced in #12536, #12793, and #12853. Done means logical plans use LogicalType, physical plans use DataType, and the listed LargeUtf8, Utf8View, LargeBinary, BinaryView, and Dictionary variants are removed as specified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100