dbt-labs / dbt-labs/dbt-adapters
[Feature] Optimize Athena insert overwrite deletions by using batch operations
- Dominant language
- Python
- Stars
- 233
- Forks
- 362
- Avg merge
- 3d 22h
- Merged PRs (30d)
- 9
Description
### Is this your first time submitting a feature request?
- [x] I have read the [expectations for open source contributors](https://docs.getdbt.com/docs/contributing/oss-expectations)
- [x] I have searched the existing issues, and I could not find an existing issue for this feature
- [x] I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion
### Describe the feature
Insert overwrite of dbt-athena deletes overlapping partitions one by one. This includes all these API calls that are made for a single entry at a time:
* Get partition metadata (to get the s3 location)
* List and delete files under the partition s3 prefix
* Drop partition
All of these can be migrated to batch operations, reducing overhead caused by individual API calls.
**In my practical test that had 450 partitions to delete, the total duration for the deletion of overlapping partitions was reduced _from 11 minutes to 1 minute 30 seconds_**.
### Describe alternatives you've considered
I don't see any clean alternative, apart from using something else than `table_type=hive` with `incremental_strategy=insert_overwrite`.
### Who will this benefit?
Anyone that is writing more than a few partitions per run.
### Are you interested in contributing this feature?
Yes, I have a working prototype
### Anything else?
First of all, this loop (which is called as part of insert ovewrite):
https://github.com/dbt-labs/dbt-adapters/blob/4b3fc1e9c11a6b454672e02bca5af186dc5caf96/dbt-athena/src/dbt/include/athena/macros/materializations/models/incremental/helpers.sql#L116-L118
can be changed to:
```sql
{%- do adapter.clean_up_partitions(target_relation, partitions) -%}
```
and then this function singature:
https://github.com/dbt-labs/dbt-adapters/blob/4b3fc1e9c11a6b454672e02bca5af186dc5caf96/dbt-athena/src/dbt/adapters/athena/impl.py#L404
is respectively changed to:
```python
def clean_up_partitions(self, relation: AthenaRelation, where_conditions: List[str]) -> None:
```
Then the `clean_up_partitions` function can be optimised as follows::
* `get_partitions` is called in chunks with multiple partition conditions with expression max length 2048 characters
* In my test case with a single partitioned column one call could fit 66 partitions.
* `objects.filter(Prefix=prefix).delete()` can be changed to:
1. calling `s3_bucket.objects.filter(Prefix=prefix)` for each prefix
2. calling `delete_objects` for the listed s3 objects across all partitions in chunks of max 1000 objects
* `delete_partition` can be replaced with `batch_delete_partition` in chunks of 25 partitions
Links:
* https://docs.aws.amazon.com/AmazonS3/latest/userguide/delete-multiple-objects.html
* https://docs.aws.amazon.com/glue/latest/webapi/API_BatchDeletePartition.html#API_BatchDeletePartition_RequestSyntax
Contributor guide
Assessment
This issue has not been assessed yet.