aws-samples / aws-samples/dbt-glue

Temporary table file format to parquet

Open
#409 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
147
Forks
96
Avg merge
7h 4m
Merged PRs (30d)
5

Description

### Describe the feature
Update the `glue__create_tmp_table_as` macro so that the file format uses parquet instead of flat file.

### Describe alternatives you've considered
I have researched methods to decrease the amount of files created as the result of an `insert overwrite` incremental operation but none have been successful, except with the change suggested above. Use of the AQE to coalesce files with a configuration option such as `spark.sql.adaptive.coalescePartitions.minPartitionSize` or `spark.sql.adaptive.coalescePartitions.minPartitionNum` will not work as the final insert operation does not use shuffle partitions.

### Additional context
Internal testing decreased the number of files created in a new partition by about 4x when the temporary table was created with parquet. Examining execution plans in SparkUI showed a lesser number of tasks generated on the scan from the temporary table during the insert into the base table. The tasks also packed in more rows versus when the temporary table was flatfile based.
Flat file based: 896 tasks w/ ~125k records each
Parquet based: 224 tasks w/ ~468k records each

### Who will this benefit?
This should benefit anyone that utilizes `insert overwrite` incrementals.

### Are you interested in contributing this feature?
I need to determine if I can submit a PR.

Contributor guide

Open the contributing guide

Research direction

Start by locating the `glue__create_tmp_table_as` macro and inspect any existing tests or callers that cover temporary-table creation. Confirm how the temporary table format is currently selected, then verify that `insert overwrite` incremental behavior uses parquet and that the relevant tests or execution checks pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
spark
Domain
data-engineering
Issue type
Feature
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.