apache / apache/arrow

[C++] Arrow ipc format supports deduplication of dictionaries with dict_id, but there is no way to exercise it

Open
#49,646 2 comments 2 reactions 0 assignees View on GitHub
Component: C++ Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the enhancement requested

Arrow ipc format assigns `dict_id` to dictionary arrays and serializes dictionaries first.
It is allowed to use the same `dict_id ` in multiple columns so you can serialize dictionary once and use multiple times.

Currently ipc writer always assigns new dict_id to every dictionary encountered. Hence, while the dictionary deduplication is supported by the ipc format, it can't be exercised by the user.

I suggest to add to `IpcWriteOptions` new option `dedup_dictionaries`. Please opine on the name of the option.
If enabled the writer will check every dictionary buffer is unique. The ipc writer will serialize only unique dictionaries to the file and reuse dict_ids.

I am interested in this feature and I can implement it if there are no objections across the dev community.

I can do C++ and Python parts.

### Component(s)

C++
Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.