apache / apache/datafusion

Reconsider equality of `ScalarValue`s for encoded types (REE/Dictionary)

Open
#20,121 6 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

@brancz brought up a point on how we might want to not consider the index/key types of REE/Dict scalarvalues when checking equality:

- https://github.com/apache/datafusion/pull/19895#discussion_r2735247027

Code reference:

https://github.com/apache/datafusion/blob/8c478e945274086edf13ea3cab89670b6368938f/datafusion/common/src/scalar/mod.rs#L727-L739

### Describe the solution you'd like

Consider ignoring the index/key types for scalarvalue equality of REE/dictionary values. What knock-on effects will this have? What does it enable for us?

### Describe alternatives you've considered

Don't do this if we prefer to stick to strict equality.

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with datafusion/common/src/scalar/mod.rs around lines 727-739 and read the discussion in PR 19895. Trace the current equality behavior for REE and dictionary ScalarValues, then assess the requested index/key-type change and its knock-on effects; done means the behavior and resulting scope are decided.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.