awslabs / awslabs/data-solutions-framework-on-aws
Add support for automatic column statistics generation in DSF catalog constructs
Open
@lmouhib is already working on this.
Since Apr 30, 2024.
enhancement
- Dominant language
- TypeScript
- Stars
- 147
- Forks
- 31
- PR merge metrics
- No merged PRs in 30d
Description
AWS native products including EMR, Athena, Glue, Redshift provide better query performances with Glue Column statistics
https://docs.aws.amazon.com/glue/latest/dg/column-statistics.html
DSF should provide an option (with default) to enable column statistic generation at the database level when creating a `DatabaseCatalog`.
DSF should create the gold layer of the `DataLakeCatalog` with column statistics enabled.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.