apache / apache/datafusion

`ScalarValue::try_from_array` holds onto input array for Struct, Lists types

Open
#15,205 3 comments 1 reaction 1 assignee Claimed by @chenkovsky View on GitHub
bug performance
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

ScalarValue::try_from_array are used in areas such as accumulators.
Currently arrow`slice` is used for Struct/Union/List types, causing accumulators holding onto the original arrays (rather than a single value), and thus higher memory usage.

### To Reproduce

Create an (struct / list type) array of size n and use `ScalarValue::try_from_array` to convert it into a `ScalarValue`.

### Expected behavior

Value is copied into a new `Array` or there would be needing efficient way of representing the scalar value.
Memory usage should be corresponding to a single value rather than the full input array.

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.