apache / apache/arrow-go

[Go] Dictionary compaction

Open
#49 6 comments 0 reactions 1 assignee Claimed by @kevinmingtarja View on GitHub
Type: enhancement
Dominant language
Assembly
Stars
404
Forks
145
Avg merge
2d 4h
Merged PRs (30d)
87

Description

### Describe the enhancement requested

I don't know if other languages offer this, but I assume this is a common thing needed.

My scenario: I have dictionary arrays that traverse a query plan and potentially get filtered, and at the end, the dictionary may contain many more items than are actually still being referenced by indices in the dictionary.

So I think it would be useful to have a `CompactDictionary` function that removes any unused entries from the dictionary and re-writes the indices to match the new, compact dictionary.

Thoughts? (We may have to implement this anyway so in a way I'm asking if anyone has already written something like this and if not if this is a contribution that would be accepted)

cc @zeroshade

### Component(s)

Go

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.