aws-samples / aws-samples/dbt-glue

Incremental merge on Iceberg tables leaves orphaned data files from temporary tables in S3

Open
#573 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
147
Forks
96
Avg merge
7h 4m
Merged PRs (30d)
5

Description

### Describe the bug
When running an **incremental** model with **file_format='iceberg'**, the temporary table, that is meant to stage the incremental data, is created and subsequently dropped as expected from the AWS Glue Data Catalog. However, the underlying data files are left behind in S3.

This occurs because the created temporary table is an **external table**. And by design, the standard `DROP TABLE` command for an external table **only** removes its metadata definition from the AWS Glue Data Catalog. It intentionally leaves the underlying data files on S3 untouched. This default behavior results in accumulating orphaned files with each dbt run.

The issue could be resolved by modifying the drop statement in the [glue__drop_relation macro](https://github.com/aws-samples/dbt-glue/blob/5e79c28d221b8cbdaac403e3abfb122e8985aca1/dbt/include/glue/macros/adapters.sql#L72C1-L87C15) to ensure the underlying data files are deleted along with the table's metadata.
`drop table if exists {{ full_relation }} purge`
Or by, somehow, changing the type of the temporary table to be managed table (the temp table can be created somewhere else other that AWS Glue as AWS Glue tables are all external tables).

We have updated the code locally, added the `PURGE` keyword, and it seems to work perfectly. However, it would need some more testing though.

### Steps To Reproduce

1. Configure a dbt project to use `dbt-glue`
2. create an incremental model with `file_format='iceberg'`
3. run the model and check both AWS Glue Data Catalog and S3.

### Expected behavior
After the dbt run successfully completes, the temporary table created during the incremental load should be completely removed. This includes both its metadata entry in the AWS Glue Data Catalog and all of its underlying data files in S3. No orphaned files should remain.

### Screenshots and log output
While there are no specific error logs (because the run completes "successfully"), the evidence of the bug can be seen by Browse the S3 bucket and observing the leftover data files from the temporary table.

### System information
**The output of `dbt --version`:**
```
Core:
- installed: 1.9.8
- latest: 1.10.3 - Update available!

Your version of dbt-core is out of date!
You can find instructions for upgrading here:
https://docs.getdbt.com/docs/installation

Plugins:
- glue: 1.9.4 - Up to date!
- spark: 1.9.2 - Up to date!
```

**The operating system you're using:**
macOS

**The output of `python --version`:**
3.10.10

### Additional context
Add any other context about the problem here.

Contributor guide

Open the contributing guide

Research direction

Read the glue__drop_relation macro in dbt/include/glue/macros/adapters.sql, then run the listed incremental model reproduction with file_format='iceberg'. Compare the AWS Glue Data Catalog and S3 before and after the dbt run; done means the temporary table metadata and its underlying data files are both removed.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud, databases
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.