Dictionary IDs Arrow IPC
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 169
Description
**Which part is this question about**
The `Field` data structure contains a `dict_id` member, that stores an i64. It appears the intention of this is that different dictionaries will have different IDs, however, this currently appears to only be respected by the IPC format and isn't widely utilised by arrow-rs.
**Describe your question**
Most of arrow-rs is completely agnostic to dict_id, with compute kernels completely ignoring it, even those that recompute dictionaries such as concat.
The only parts of the stack that appear to use the dict_ids are the IPC interfaces, which will error if they encounter the same dict_id multiple times. I think this is inconsistency is a tad confusing, I think we should do one of the following:
* Keep the current agnosticism within arrow-rs and assign IDs in the writers (potentially using Arc::ptr_eq on the values array)
* Make arrow-rs respect dict_ids
Of these the first would definitely be simpler to implement, but I'm not familiar enough with the purpose of dict_id to be certain there isn't some use-case this would preclude?
**Additional context**
As `Field` is part of the `Schema`, RecordBatch with different dict_id will appear to have different schema. This may have downstream implications for things like DataFusion which have strong assumptions on schema consistency within a plan.
This cropped up in https://github.com/apache/arrow-datafusion/pull/1596 as it is using the arrow IPC format to spill buffers to disk.
Contributor guide
Research direction
Read the Field dict_id semantics and the IPC interfaces first, then compare how compute kernels and concat handle dictionary IDs. Use the linked DataFusion pull request as context; done requires choosing and documenting one consistent ownership model and applying the resulting behavior across the affected paths.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100