Leverage dictionary-encode when turning a scalar columnar value into an array
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
We have a `into_array` function of `ColumnarValue` which converts it into an arrow array like this:
```rust
pub fn into_array(self, num_rows: usize) -> Result {
Ok(match self {
ColumnarValue::Array(array) => array,
ColumnarValue::Scalar(scalar) => scalar.to_array_of_size(num_rows)?,
})
}
```
If this column is of an Int32 type, it returns an Int32Array.
Given that the `ColumnarValue::Scalar`'s cardinality is 1, in some special cases, we can turn it into a dictionary array in order to obtain some performance gains.
### Describe the solution you'd like
For example, if we want to turn a scalar value of f64 type into an array of size 256, we can return a dictionary f64 array of uint8 key type.
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
Contributor guide
Research direction
Start at ColumnarValue::into_array and follow the Scalar path through scalar.to_array_of_size. Determine how a scalar value can become a dictionary array with an appropriate key type while preserving the requested row count and array semantics; done means scalar conversion can use this representation where applicable.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100