apache / apache/arrow-go

[Go] Allow prepending dictionary

Open
#47 6 comments 0 reactions 1 assignee Claimed by @brancz View on GitHub
Type: enhancement
Dominant language
Assembly
Stars
404
Forks
145
Avg merge
2d 4h
Merged PRs (30d)
87

Description

### Describe the enhancement requested

The dictionary builders already have methods to insert whole arrays, but unfortunately they cause a lot of potentially unnecessary CPU time.

Take the following scenario: I have two sources of data, one of them is already dictionary encoded, the other is not, so I would like to initialize the dictionary builder with the existing dictionary, and only insert new items for the non-dictionary encodede items. Now comes the important part: I'm ok with inserts potentially creating duplicates in the dictionary.

I would like to propose a new API `PrependInitialDict`, that takes an array and *must* be called before inserting into the indices array, otherwise it errors, and then any new dictionary item inserted start at `len(initialDict)+i`.

Theoretically it could even be designed to insert dicts multiple times, but I would suggest to start the API like this and only extend when we have the use cases.

---

Alternative I have considered: Prepending the dictionary after building the "new" dictionary and have any indices start at the length. I've found this to not really be workable, for two reasons:

1) There would still have to be an API to set the initial index.
2) It would rely on the user actually prepending the dictionary afterward (easy to misuse).
3) It would be quite awkward to use in scenarios where there are deeply nested lists and structs, where building the final record is primarily done using a record builder, but only this array would be the exception.

cc @zeroshade

### Component(s)

Go

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.