aws-samples / aws-samples/dbt-glue

Add bucketing for file_format parquet

Open
#100 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
147
Forks
96
Avg merge
7h 4m
Merged PRs (30d)
5

Description

### Describe the feature
Add the possibility to specify bucketing with file_format=parquet to improve performance on reading.

### Describe alternatives you've considered
Nothing that come to mind, bucketing is different than partitioning.

### Additional context
Bucketing can speed up queries when working with high cardinality fields, it's a must have.

### Who will this benefit?
All the use cases where specific WHERE conditions are used on the bucketed columns (e.g. id/user_id), in general bucketing is recommended for high cardinality columns.

### Are you interested in contributing this feature?
Yup.

Contributor guide

Open the contributing guide

Research direction

No files, tests, or entry points are named. Start by locating the adapter's handling of file_format=parquet and compare it with the existing partitioning behavior; done should mean bucketing can be specified for Parquet and improves reads for filters on bucketed columns.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.