apache / apache/arrow-rs

[ipc] Create an integrated StreamWriter and RecordBatchBuilder to enable efficient delta dictionary emission

Open
#8,134 2 comments 2 reactions 0 assignees View on GitHub
arrow enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 14h
Merged PRs (30d)
167

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

This is an issue to capture the conversations [here](https://github.com/apache/arrow-rs/pull/8001) about ways to improve the efficiency of delta dictionary emission in the future.

There is discussion in that thread about the conditions under which we can emit delta dictionaries and some efficiency concerns with doing that which I think stems mostly from separation of the record batch builder from the ipc writer itself.

**Describe the solution you'd like**

One idea I had is roughly describe [here](https://github.com/apache/arrow-rs/pull/8001#issuecomment-3184791114).

I'll copy paste the most relevant part here which is that we could:

1. Have an IPC stream writer which owns the record batch building rather than just having finished record batches pushed into it.
2. Add APIs to the dictionary builders to track and emit the incremental deltas for whatever they're building intsead of the full set.

That way the IPC writer will know for sure that every time it seals a batch that it has the exact delta for every dictionary since it owns the record batch production and is controlling what the dictionary builders do.

**Describe alternatives you've considered**

We could also have the IPC writer re-intern the dictionaries and get good deltas that way, but that would require a lot of extra hashing and be expensive.

**Additional context**

CC: @alamb @albertlockett

Contributor guide

Open the contributing guide

Research direction

Start by reading the discussion in linked pull request 8001, then inspect the IPC stream writer, record batch builder, and dictionary builder APIs it references. Define the ownership and delta-emission behavior before identifying the affected Rust modules and tests; done means the integrated flow can emit exact dictionary deltas without re-interning.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.