apache / apache/arrow-rs

Compact ViewArrays on IPCWrite

Open
#7,185 7 comments 0 reactions 0 assignees View on GitHub
arrow enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 16h
Merged PRs (30d)
168

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

The IPC writer typically tries to avoid writing unreferenced data, e.g. re-encoding offsets for StringArray, ListArray, etc... see `write_array_data`.

Currently the IPC writer writes view data types as is, potentially including significant unreferenced data.

**Describe the solution you'd like**

I think by default the IPC writer should at least invoke the [GC](https://docs.rs/arrow-array/latest/arrow_array/array/struct.GenericByteViewArray.html#method.gc) API, and potentially [deduplicate](https://docs.rs/arrow-array/latest/arrow_array/builder/struct.GenericByteViewBuilder.html#method.with_deduplicate_strings) them. We could potentially make this configurable.

**Describe alternatives you've considered**

**Additional context**

Potentially some overlap with
- #7186
- https://github.com/apache/arrow-rs/issues/7184

FYI @alamb @XiangpengHao

Contributor guide

Open the contributing guide

Research direction

Start with the IPC writer's write_array_data path and the GenericByteViewArray::gc API mentioned in the issue. Determine how view data is currently serialized, then verify that IPC output omits unreferenced data while considering whether string deduplication or configurability is in scope.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.