[C++] Arrow ipc format supports deduplication of dictionaries with dict_id, but there is no way to exercise it
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the enhancement requested
Arrow ipc format assigns `dict_id` to dictionary arrays and serializes dictionaries first.
It is allowed to use the same `dict_id ` in multiple columns so you can serialize dictionary once and use multiple times.
Currently ipc writer always assigns new dict_id to every dictionary encountered. Hence, while the dictionary deduplication is supported by the ipc format, it can't be exercised by the user.
I suggest to add to `IpcWriteOptions` new option `dedup_dictionaries`. Please opine on the name of the option.
If enabled the writer will check every dictionary buffer is unique. The ipc writer will serialize only unique dictionaries to the file and reuse dict_ids.
I am interested in this feature and I can implement it if there are no objections across the dev community.
I can do C++ and Python parts.
### Component(s)
C++
Python
Contributor guide
Assessment
This issue has not been assessed yet.