dbt-labs / dbt-labs/dbt-adapters

[Bug] BigQuery docs generate treats region-scoped INFORMATION_SCHEMA sources as physical datasets

Open
#2,152 0 comments 0 reactions 0 assignees View on GitHub
triage:product
Dominant language
Python
Stars
233
Forks
362
Avg merge
3d 22h
Merged PRs (30d)
9

Description

### Is this a new bug?

- [x] I believe this is a new bug
- [x] I have searched the existing issues and could not find a duplicate

### Which packages are affected?

- [x] dbt-bigquery
- [x] dbt-adapters

### Current Behavior

With `dbt-core==1.12.0` and `dbt-bigquery==1.12.0`, `dbt docs generate`
fails when a selected source uses a region-scoped BigQuery `INFORMATION_SCHEMA`
qualifier:

```yaml
version: 2

sources:
- name: information_schema
database: my-project
schema: region-eu.INFORMATION_SCHEMA
tables:
- name: tables
identifier: TABLES
```

The adapter calls the BigQuery REST `tables.list` endpoint as though
`region-eu.INFORMATION_SCHEMA` were a physical dataset:

```text
403 GET https://bigquery.googleapis.com/bigquery/v2/projects//datasets/region-eu.INFORMATION_SCHEMA/tables?maxResults=1

Access Denied: Namespace .region-eu.INFORMATION_SCHEMA:
Permission biglake.tables.list denied on namespace
.region-eu.INFORMATION_SCHEMA
```

The relevant call path is:

```text
get_filtered_catalog
-> get_catalog_by_relations
-> BigQueryAdapter._get_catalog_relations_by_info_schema
-> check_schema_exists
-> client.list_tables
```

The exception aborts catalog generation and `target/catalog.json` is not
written.

`region-eu.INFORMATION_SCHEMA` is a region-scoped query qualifier, not a
physical BigQuery dataset. Calling `datasets.tables.list` with that identifier
causes it to be authorized as a BigLake namespace.

### Expected Behavior

`dbt docs generate` should support projects that define region-scoped
`INFORMATION_SCHEMA` sources without treating those source schemas as physical
datasets.

The adapter should skip the physical dataset existence check for this form of
relation, or otherwise prevent that check from aborting catalog generation.
The remaining selected models and sources should still be written to
`catalog.json`.

This appears to be an uncovered edge case in the fix for #1005 / #1940. That
issue uses the same `region-.INFORMATION_SCHEMA` source pattern, and
its expected behavior is that the source should not collide with catalog
generation.

### Steps To Reproduce

1. Configure a BigQuery profile.
2. Add the source definition shown above.
3. Run `dbt docs generate`.
4. Observe the 403 request against
`datasets/region-eu.INFORMATION_SCHEMA/tables`.
5. Confirm that `target/catalog.json` is not created.

### Relevant log output

```text
[ERROR]: Encountered an error:
403 GET https://bigquery.googleapis.com/bigquery/v2/projects//datasets/region-eu.INFORMATION_SCHEMA/tables?maxResults=1
Access Denied: Namespace .region-eu.INFORMATION_SCHEMA:
Permission biglake.tables.list denied on namespace
.region-eu.INFORMATION_SCHEMA
```

### Environment

- OS: Linux container running in Kubernetes
- Python: 3.12
- dbt-core: 1.12.0
- dbt-bigquery: 1.12.0
- BigQuery location: EU

### Additional Context

A targeted application-level workaround is to exclude these sources from
catalog generation:

```shell
dbt docs generate --exclude source:information_schema
```

Granting `roles/biglake.viewer` does not seem like an appropriate general
workaround because the selected relation is an `INFORMATION_SCHEMA`
pseudo-dataset rather than a BigLake table or namespace.

Related:

- https://github.com/dbt-labs/dbt-adapters/issues/1005
- https://github.com/dbt-labs/dbt-adapters/pull/1940

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.