apache / apache/arrow-rs

Dictionary IDs Arrow IPC

Open
#1,206 8 comments 0 reactions 0 assignees View on GitHub
question
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

**Which part is this question about**

The `Field` data structure contains a `dict_id` member, that stores an i64. It appears the intention of this is that different dictionaries will have different IDs, however, this currently appears to only be respected by the IPC format and isn't widely utilised by arrow-rs.

**Describe your question**

Most of arrow-rs is completely agnostic to dict_id, with compute kernels completely ignoring it, even those that recompute dictionaries such as concat.

The only parts of the stack that appear to use the dict_ids are the IPC interfaces, which will error if they encounter the same dict_id multiple times. I think this is inconsistency is a tad confusing, I think we should do one of the following:

* Keep the current agnosticism within arrow-rs and assign IDs in the writers (potentially using Arc::ptr_eq on the values array)
* Make arrow-rs respect dict_ids

Of these the first would definitely be simpler to implement, but I'm not familiar enough with the purpose of dict_id to be certain there isn't some use-case this would preclude?

**Additional context**

As `Field` is part of the `Schema`, RecordBatch with different dict_id will appear to have different schema. This may have downstream implications for things like DataFusion which have strong assumptions on schema consistency within a plan.

This cropped up in https://github.com/apache/arrow-datafusion/pull/1596 as it is using the arrow IPC format to spill buffers to disk.

Contributor guide

Open the contributing guide

Research direction

Read the Field dict_id semantics and the IPC interfaces first, then compare how compute kernels and concat handle dictionary IDs. Use the linked DataFusion pull request as context; done requires choosing and documenting one consistent ownership model and applying the resulting behavior across the affected paths.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.