Reconsider equality of `ScalarValue`s for encoded types (REE/Dictionary)
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
@brancz brought up a point on how we might want to not consider the index/key types of REE/Dict scalarvalues when checking equality:
- https://github.com/apache/datafusion/pull/19895#discussion_r2735247027
Code reference:
https://github.com/apache/datafusion/blob/8c478e945274086edf13ea3cab89670b6368938f/datafusion/common/src/scalar/mod.rs#L727-L739
### Describe the solution you'd like
Consider ignoring the index/key types for scalarvalue equality of REE/dictionary values. What knock-on effects will this have? What does it enable for us?
### Describe alternatives you've considered
Don't do this if we prefer to stick to strict equality.
### Additional context
_No response_
Contributor guide
Research direction
Start with datafusion/common/src/scalar/mod.rs around lines 727-739 and read the discussion in PR 19895. Trace the current equality behavior for REE and dictionary ScalarValues, then assess the requested index/key-type change and its knock-on effects; done means the behavior and resulting scope are decided.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 32/100