dlt-hub / dlt-hub/dlt

Fix support for distinct auth/billing and destination projects in BigQuery

Open
#4,160 1 comment 0 reactions 1 assignee Claimed by @Travior View on GitHub
bug destination needs decision
Dominant language
Python
Stars
5.9k
Forks
600
Avg merge
1d 14h
Merged PRs (30d)
38

Description

### dlt version

1.27.2

### Describe the problem

The documentation makes it seem like it's possible to set separate project IDs for auth (and thus billing) and destination: https://dlthub.com/docs/dlt-ecosystem/destinations/bigquery#using-different-project_id

> # Using different project_id
> You can set the project_id in your configuration to be different from the one in your credentials, provided your account has access to it:
> ```
> [destination.bigquery]
> project_id = "project_id_destination"
>
> [destination.bigquery.credentials]
> project_id = "project_id_credentials"
> ```
> In this scenario, project_id_credentials will be used for authentication, while project_id_destination will be used as the data destination.

However, the BigQuery implementation doesn't support it: https://github.com/dlt-hub/dlt/blob/devel/dlt/destinations/impl/bigquery/sql_client.py#L96
```python
self.project_id = project_id or self.credentials.project_id
```

### Expected behavior

Using the config described in the docs actually works, and different

### Steps to reproduce

Set up 2 BigQuery projects, having `roles/bigquery.jobUser` in only one of them (e.g. project-a), while setting the other as the destination project (e.g. project-b).

Expected:
- dlt should run and the execution should be billed to project-a, while data should be inserted in the datasets residing in project-b.

Actual:
- error: `403 POST https://bigquery.googleapis.com/bigquery/v2/projects//queries?prettyPrint=false: Access Denied: Project : User does not have bigquery.jobs.create permission in project `.

### Operating system

Linux

### Runtime environment

Other

### Python version

3.13

### dlt data source

API

### dlt destination

Google BigQuery

### Other deployment details

Running in GCP CloudRun

### Additional information

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.