apache / apache/datafusion

[EPIC] Decouple logical from physical types

Open
#12,622 39 comments 0 reactions 0 assignees View on GitHub
enhancement PROPOSAL EPIC
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

This epic tracks an ordered list tasks related to the proposal: Decouple logical from physical types (#11513). The goal is:

> Logical operators during logical planning should unquestionably **not** have access to the physical type information, which should exclusively be reserved to the physical planning and physical execution.
>
> LogicalPlans will use LogicalType while PhysicalPlans will use DataType.

### Describe the solution you'd like

Make `ScalarValue` values logical:

- [x] #12536 (merged in `logical-types`)
- [x] #12793 (merged in `logical-types`)
- [x] #12853
- [ ] Remove `ScalarValue::LargeUtf8 `and `ScalarValue::Utf8View` in favour of `ScalarValue::Utf8`
- [ ] Remove `ScalarValue::LargeBinary` and `ScalarValue::BinaryView` in favour of `ScalarValue::Binary`
- [ ] Remove `ScalarValue::Dictionary` (from https://github.com/apache/datafusion/pull/12488)

Contributor guide

Open the contributing guide

Research direction

Review the proposal in #11513 and the remaining unchecked ScalarValue tasks, starting with the logical-types work referenced in #12536, #12793, and #12853. Done means logical plans use LogicalType, physical plans use DataType, and the listed LargeUtf8, Utf8View, LargeBinary, BinaryView, and Dictionary variants are removed as specified.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.