BigQuery: dataset existence check lists every dataset in the project (including hidden ones), which may be slow at scale
- Dominant language
- Rust
- Stars
- 13.8k
- Forks
- 2.6k
- Avg merge
- 21h 31m
- Merged PRs (30d)
- 56
Description
### Is this a new bug in dbt v2.x compared to the latest version of dbt 1.x?
- [x] I believe this is a new bug in dbt v2.x
- [x] I have searched the existing issues and could not find a duplicate
### Current Behavior
On dbt Core 2.0.0-rc.2, BigQuery adapter, Linux x86_64, run, seed and compile appear to verify the target dataset by listing all datasets (includes hidden) in the target project before the first query is issued. So its duration grows with the total number of datasets in the project.
BigQuery projects accumulate hidden anonymous datasets (the _… datasets created per principal for cached query results), and in a project where many users or service accounts have run queries over time these can outnumber the visible datasets by a large factor. Because they are included in the listing, the delay before the first query scales with that count. dbt debug is unaffected because it does not perform the check.
This is invisible for projects, where the dataset count is in the hundreds, but becomes significant for projects with a large number of datasets or many distinct query principals.
What is observed
- Requests are sequential at roughly two per second, so the pre-query delay is about N / 50 seconds for N datasets: a project with 10,000 datasets would wait over three minutes per invocation.
- The behaviour is the same for run, seed and compile, and independent of --threads, --static-analysis off, or project size.
Where it seems to come from
list_schemas for BigQuery goes through list_schemas_via_adbc, which calls GetObjects at schema depth for the whole catalog with no schema filter (the code carries a TODO about adding one). The check appears to have been introduced so that CREATE SCHEMA IF NOT EXISTS is only attempted when the dataset is missing (#14631).
### Expected Behavior
dbt Core 1.x checks the single target dataset with a datasets.get call, so the cost there is constant regardless of how many datasets the project contains.
### Steps To Reproduce
1. Use a BigQuery project that contains a large number of datasets, including hidden anonymous datasets (any project where many users or service accounts have run queries over time will have many _… datasets; bq ls -a or Client.list_datasets(include_all=True) shows them).
2. Create a minimal dbt project with one seed and one model that selects from it, and a profile pointing at a dataset in that project.
3. Run dbt seed or dbt run with GODEBUG=http2debug=1 set.
4. Observe the sequence of GET /bigquery/v2/projects//datasets?all=true&pageToken=… requests before any job is submitted, and the elapsed time before the first query.
### Relevant log output
```shell
```
### Environment
```markdown
- OS: Linux x86_64
- CPU: x86_64
- dbt distribution and version: (`dbt --version`) Core 2.0.0-rc.2
```
### Which database adapter are you using?
bigquery
### Is this a discrepancy vs. dbt 1.x?
- [x] Yes — this works in dbt 1.x but not in dbt v2.x
### Additional Context
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.