dbt-labs / dbt-labs/dbt-adapters
[Feature] support copy multiple tables in parallel using copy_partitions
- Dominant language
- Python
- Stars
- 233
- Forks
- 362
- Avg merge
- 3d 22h
- Merged PRs (30d)
- 9
Description
### Is this your first time submitting a feature request?
- [X] I have read the [expectations for open source contributors](https://docs.getdbt.com/docs/contributing/oss-expectations)
- [X] I have searched the existing issues, and I could not find an existing issue for this feature
- [X] I am requesting a straightforward extension of existing dbt-bigquery functionality, rather than a Big Idea better suited to a discussion
### Describe the feature
Python BigQuery Client supports asynchronous [copy jobs](https://github.com/googleapis/python-bigquery/blob/a86d7b96813f67fea28b46c5252416222edca9a6/google/cloud/bigquery/job/copy_.py#L146) while the dbt-bigquery adapter sends BigQuery requests one by one (using `incremental_strategy = 'insert_overwrite'` with `copy_partitions=true`).
We can achieve a better performance if we start sending requests in small batches of partitions.
dbt-bigquery already supports parallel execution in the [copy_bq_table](https://github.com/dbt-labs/dbt-bigquery/blob/b7bb0bc0818644617b8c10eb0723560ce98147a5/dbt/adapters/bigquery/connections.py#L621) function.
But in the [bq_copy_partitions](https://github.com/dbt-labs/dbt-bigquery/blob/b7bb0bc0818644617b8c10eb0723560ce98147a5/dbt/include/bigquery/macros/materializations/incremental_strategy/insert_overwrite.sql#L19
) macro partitions are sent one at a time.
We can probably implement this feature by introducing a `batch_size` argument to the configs:
```sql
{{ config(
materialized = 'incremental',
incremental_strategy = 'insert_overwrite',
partition_by = {
"field": "day",
"data_type": "date",
"copy_partitions": true,
"batch_size": 5
}
) }}
```
Default value will be 1. And `bq_copy_partitions` macro will send a list of partitions to the `copy_bq_table`, where the size of list = `batch_size`.
### Describe alternatives you've considered
_No response_
### Who will this benefit?
Anyone who has high amount of heavy BigQuery partitions.
### Are you interested in contributing this feature?
Definitely, just need a green light to proceed
### Anything else?
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.