dbt-labs / dbt-labs/dbt-adapters

[Feature] support copy multiple tables in parallel using copy_partitions

Open
#559 1 comment 12 reactions 0 assignees View on GitHub
feature:python-models pkg:dbt-bigquery type:enhancement
Dominant language
Python
Stars
233
Forks
362
Avg merge
3d 22h
Merged PRs (30d)
9

Description

### Is this your first time submitting a feature request?

- [X] I have read the [expectations for open source contributors](https://docs.getdbt.com/docs/contributing/oss-expectations)
- [X] I have searched the existing issues, and I could not find an existing issue for this feature
- [X] I am requesting a straightforward extension of existing dbt-bigquery functionality, rather than a Big Idea better suited to a discussion

### Describe the feature

Python BigQuery Client supports asynchronous [copy jobs](https://github.com/googleapis/python-bigquery/blob/a86d7b96813f67fea28b46c5252416222edca9a6/google/cloud/bigquery/job/copy_.py#L146) while the dbt-bigquery adapter sends BigQuery requests one by one (using `incremental_strategy = 'insert_overwrite'` with `copy_partitions=true`).

We can achieve a better performance if we start sending requests in small batches of partitions.

dbt-bigquery already supports parallel execution in the [copy_bq_table](https://github.com/dbt-labs/dbt-bigquery/blob/b7bb0bc0818644617b8c10eb0723560ce98147a5/dbt/adapters/bigquery/connections.py#L621) function.
But in the [bq_copy_partitions](https://github.com/dbt-labs/dbt-bigquery/blob/b7bb0bc0818644617b8c10eb0723560ce98147a5/dbt/include/bigquery/macros/materializations/incremental_strategy/insert_overwrite.sql#L19
) macro partitions are sent one at a time.

We can probably implement this feature by introducing a `batch_size` argument to the configs:
```sql
{{ config(
materialized = 'incremental',
incremental_strategy = 'insert_overwrite',
partition_by = {
"field": "day",
"data_type": "date",
"copy_partitions": true,
"batch_size": 5
}
) }}
```

Default value will be 1. And `bq_copy_partitions` macro will send a list of partitions to the `copy_bq_table`, where the size of list = `batch_size`.

### Describe alternatives you've considered

_No response_

### Who will this benefit?

Anyone who has high amount of heavy BigQuery partitions.

### Are you interested in contributing this feature?

Definitely, just need a green light to proceed

### Anything else?

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.