apache / apache/parquet-format
Store `dictionary entries` of parquet columns that will be used for joins
- Dominant language
- Thrift
- Stars
- 2.6k
- Forks
- 508
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 4
Description
It would be great if Parquet would store `dictionary entries` for columns marked to be used for joins.
When a column is used for a join (it could be a [surrogate key](https://en.wikipedia.org/wiki/Surrogate_key) or a [natural key](https://en.wikipedia.org/wiki/Natural_key)) - the value of a cloumn used for join itself is actually not so important.
So we could join directly on `dictionary entries` instead of values
and save CPU cycles. (no need to decompress etc)
Inspired by [Oracle In-memory columnar storage improvements in 12.2](https://blogs.oracle.com/In-Memory/entry/what_s_new_in_12)
**Reporter**: [Ruslan Dautkhanov](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=Tagar) / @Tagar
**Note**: *This issue was originally created as [PARQUET-966](https://issues.apache.org/jira/browse/PARQUET-966). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*
Contributor guide
Research direction
The issue names no files, tests, or entry points. It proposes storing dictionary entries for join columns, but provides no implementation location or acceptance criteria.
Written by the indexing model from the issue text.
Assessment
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100