Compact ViewArrays on IPCWrite
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 168
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
The IPC writer typically tries to avoid writing unreferenced data, e.g. re-encoding offsets for StringArray, ListArray, etc... see `write_array_data`.
Currently the IPC writer writes view data types as is, potentially including significant unreferenced data.
**Describe the solution you'd like**
I think by default the IPC writer should at least invoke the [GC](https://docs.rs/arrow-array/latest/arrow_array/array/struct.GenericByteViewArray.html#method.gc) API, and potentially [deduplicate](https://docs.rs/arrow-array/latest/arrow_array/builder/struct.GenericByteViewBuilder.html#method.with_deduplicate_strings) them. We could potentially make this configurable.
**Describe alternatives you've considered**
**Additional context**
Potentially some overlap with
- #7186
- https://github.com/apache/arrow-rs/issues/7184
FYI @alamb @XiangpengHao
Contributor guide
Research direction
Start with the IPC writer's write_array_data path and the GenericByteViewArray::gc API mentioned in the issue. Determine how view data is currently serialized, then verify that IPC output omits unreferenced data while considering whether string deduplication or configurability is in scope.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100