apache / apache/datafusion

Leverage dictionary-encode when turning a scalar columnar value into an array

Open
#11,503 3 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

We have a `into_array` function of `ColumnarValue` which converts it into an arrow array like this:
```rust
pub fn into_array(self, num_rows: usize) -> Result {
Ok(match self {
ColumnarValue::Array(array) => array,
ColumnarValue::Scalar(scalar) => scalar.to_array_of_size(num_rows)?,
})
}
```
If this column is of an Int32 type, it returns an Int32Array.
Given that the `ColumnarValue::Scalar`'s cardinality is 1, in some special cases, we can turn it into a dictionary array in order to obtain some performance gains.

### Describe the solution you'd like

For example, if we want to turn a scalar value of f64 type into an array of size 256, we can return a dictionary f64 array of uint8 key type.

### Describe alternatives you've considered

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start at ColumnarValue::into_array and follow the Scalar path through scalar.to_array_of_size. Determine how a scalar value can become a dictionary array with an appropriate key type while preserving the requested row count and array semantics; done means scalar conversion can use this representation where applicable.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.