apache / apache/arrow-nanoarrow

Support dictionary-encoded types in the IPC writer

Open
#622 6 comments 2 reactions 0 assignees View on GitHub
Dominant language
C
Stars
249
Forks
68
Avg merge
2d 16h
Merged PRs (30d)
12

Description

Neither the IPC reader nor the IPC writer support dictionary encoding. The lack of writer support is problematic for the R bindings because the `factor` type is converted by default to a dictionary-encoded string; the lack of reader support is problematic because the IPC reader is currently forced to error for streams produced elsewhere, many of which will happily emit dictionary messages. This also complicates our integration testing since currently we have to skip any file that includes a dictionary.

Delta dictionaries are probably out of scope for the initial level of support.

edit: Read support is released; retitling to track write support

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the IPC writer entry point and reviewing the released dictionary support in the IPC reader. Add initial support for writing non-delta dictionary messages, then run the relevant IPC integration tests and confirm dictionary-containing files no longer need to be skipped.

Written by the indexing model from the issue text.

Assessment

Tech stack
c
Domain
data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.