apache / apache/arrow-nanoarrow
Support dictionary-encoded types in the IPC writer
- Dominant language
- C
- Stars
- 249
- Forks
- 68
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 12
Description
Neither the IPC reader nor the IPC writer support dictionary encoding. The lack of writer support is problematic for the R bindings because the `factor` type is converted by default to a dictionary-encoded string; the lack of reader support is problematic because the IPC reader is currently forced to error for streams produced elsewhere, many of which will happily emit dictionary messages. This also complicates our integration testing since currently we have to skip any file that includes a dictionary.
Delta dictionaries are probably out of scope for the initial level of support.
edit: Read support is released; retitling to track write support
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating the IPC writer entry point and reviewing the released dictionary support in the IPC reader. Add initial support for writing non-delta dictionary messages, then run the relevant IPC integration tests and confirm dictionary-containing files no longer need to be skipped.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c
- Domain
- data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100