Fix support for distinct auth/billing and destination projects in BigQuery
- Dominant language
- Python
- Stars
- 5.9k
- Forks
- 600
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 38
Description
### dlt version
1.27.2
### Describe the problem
The documentation makes it seem like it's possible to set separate project IDs for auth (and thus billing) and destination: https://dlthub.com/docs/dlt-ecosystem/destinations/bigquery#using-different-project_id
> # Using different project_id
> You can set the project_id in your configuration to be different from the one in your credentials, provided your account has access to it:
> ```
> [destination.bigquery]
> project_id = "project_id_destination"
>
> [destination.bigquery.credentials]
> project_id = "project_id_credentials"
> ```
> In this scenario, project_id_credentials will be used for authentication, while project_id_destination will be used as the data destination.
However, the BigQuery implementation doesn't support it: https://github.com/dlt-hub/dlt/blob/devel/dlt/destinations/impl/bigquery/sql_client.py#L96
```python
self.project_id = project_id or self.credentials.project_id
```
### Expected behavior
Using the config described in the docs actually works, and different
### Steps to reproduce
Set up 2 BigQuery projects, having `roles/bigquery.jobUser` in only one of them (e.g. project-a), while setting the other as the destination project (e.g. project-b).
Expected:
- dlt should run and the execution should be billed to project-a, while data should be inserted in the datasets residing in project-b.
Actual:
- error: `403 POST https://bigquery.googleapis.com/bigquery/v2/projects//queries?prettyPrint=false: Access Denied: Project : User does not have bigquery.jobs.create permission in project `.
### Operating system
Linux
### Runtime environment
Other
### Python version
3.13
### dlt data source
API
### dlt destination
Google BigQuery
### Other deployment details
Running in GCP CloudRun
### Additional information
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.