aws-samples / aws-samples/dbt-glue
Add bucketing for file_format parquet
- Dominant language
- Python
- Stars
- 147
- Forks
- 96
- Avg merge
- 7h 4m
- Merged PRs (30d)
- 5
Description
### Describe the feature
Add the possibility to specify bucketing with file_format=parquet to improve performance on reading.
### Describe alternatives you've considered
Nothing that come to mind, bucketing is different than partitioning.
### Additional context
Bucketing can speed up queries when working with high cardinality fields, it's a must have.
### Who will this benefit?
All the use cases where specific WHERE conditions are used on the bucketed columns (e.g. id/user_id), in general bucketing is recommended for high cardinality columns.
### Are you interested in contributing this feature?
Yup.
Contributor guide
Research direction
No files, tests, or entry points are named. Start by locating the adapter's handling of file_format=parquet and compare it with the existing partitioning behavior; done should mean bucketing can be specified for Parquet and improves reads for filters on bucketed columns.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100