dbt-labs / dbt-labs/dbt-adapters

[Feature] Optimize Athena insert overwrite deletions by using batch operations

Open
#1,125 4 comments 2 reactions 0 assignees View on GitHub
pkg:dbt-athena type:enhancement
Dominant language
Python
Stars
233
Forks
362
Avg merge
3d 22h
Merged PRs (30d)
9

Description

### Is this your first time submitting a feature request?

- [x] I have read the [expectations for open source contributors](https://docs.getdbt.com/docs/contributing/oss-expectations)
- [x] I have searched the existing issues, and I could not find an existing issue for this feature
- [x] I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

### Describe the feature

Insert overwrite of dbt-athena deletes overlapping partitions one by one. This includes all these API calls that are made for a single entry at a time:
* Get partition metadata (to get the s3 location)
* List and delete files under the partition s3 prefix
* Drop partition

All of these can be migrated to batch operations, reducing overhead caused by individual API calls.

**In my practical test that had 450 partitions to delete, the total duration for the deletion of overlapping partitions was reduced _from 11 minutes to 1 minute 30 seconds_**.

### Describe alternatives you've considered

I don't see any clean alternative, apart from using something else than `table_type=hive` with `incremental_strategy=insert_overwrite`.

### Who will this benefit?

Anyone that is writing more than a few partitions per run.

### Are you interested in contributing this feature?

Yes, I have a working prototype

### Anything else?

First of all, this loop (which is called as part of insert ovewrite):

https://github.com/dbt-labs/dbt-adapters/blob/4b3fc1e9c11a6b454672e02bca5af186dc5caf96/dbt-athena/src/dbt/include/athena/macros/materializations/models/incremental/helpers.sql#L116-L118

can be changed to:

```sql
{%- do adapter.clean_up_partitions(target_relation, partitions) -%}
```

and then this function singature:

https://github.com/dbt-labs/dbt-adapters/blob/4b3fc1e9c11a6b454672e02bca5af186dc5caf96/dbt-athena/src/dbt/adapters/athena/impl.py#L404

is respectively changed to:

```python
def clean_up_partitions(self, relation: AthenaRelation, where_conditions: List[str]) -> None:
```

Then the `clean_up_partitions` function can be optimised as follows::
* `get_partitions` is called in chunks with multiple partition conditions with expression max length 2048 characters
* In my test case with a single partitioned column one call could fit 66 partitions.
* `objects.filter(Prefix=prefix).delete()` can be changed to:
1. calling `s3_bucket.objects.filter(Prefix=prefix)` for each prefix
2. calling `delete_objects` for the listed s3 objects across all partitions in chunks of max 1000 objects
* `delete_partition` can be replaced with `batch_delete_partition` in chunks of 25 partitions

Links:
* https://docs.aws.amazon.com/AmazonS3/latest/userguide/delete-multiple-objects.html
* https://docs.aws.amazon.com/glue/latest/webapi/API_BatchDeletePartition.html#API_BatchDeletePartition_RequestSyntax

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.