dbt-labs / dbt-labs/dbt-adapters

[Feature] Continue BigQuery jobs across dbt invocations

Open
#2,070 0 comments 0 reactions 0 assignees View on GitHub
triage:product
Dominant language
Python
Stars
233
Forks
362
Avg merge
3d 22h
Merged PRs (30d)
9

Description

### Is this your first time submitting a feature request?

- [x] I have read the [expectations for open source contributors](https://docs.getdbt.com/docs/contributing/oss-expectations)
- [x] I have searched the existing issues, and I could not find an existing issue for this feature
- [x] I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

### Describe the feature

Allow a later `dbt-bigquery` invocation to attach to existing BigQuery jobs and continue the same logical orchestrator task.

PR #2054 made query and copy job submission idempotent within one dbt process by retaining a generated job ID across client retries and attaching on `409 Conflict`. A new dbt process still generates new IDs, however. If an orchestrator retries a task on another worker after the original dbt process exits while its BigQuery job continues, the retry can submit duplicate work.

The proposed opt-in behavior is:

- Accept a runtime-only BigQuery profile credential
- Derive stable job IDs for query and copy operations from the logical key plus node, microbatch, and statement/copy identity
- Attach to matching jobs created by a prior dbt invocation
- Validate a request fingerprint before attaching, and fail closed if SQL or semantic job configuration differs
- Return ordinary adapter results so the new invocation can continue dbt's materialization control flow after the previously submitted job

The runtime-only credential should be omitted from `BigQueryCredentials._connection_keys()`. Since dbt-core #9844 hashes only `connection_info()` for partial parsing, this keeps a per-run orchestration value out of `{{ target }}` and avoids invalidating partial parsing.

### Describe alternatives you've considered

- A fixed adapter environment variable is parse-neutral but bypasses typed profile validation and is awkward for programmatic callers.
- A new dbt-core CLI flag would introduce an adapter-specific Core interface; adapters do not currently expose a CLI-extension hook.

### Who will this benefit?

Users running dbt under Airflow, Kubernetes, or another distributed orchestrator where an automatic retry may execute in a fresh process or on a different machine. It prevents duplicate BigQuery work and lets the retry wait for and consume the original job's result.

### Are you interested in contributing this feature?

Yes. I can contribute the implementation, unit coverage, BigQuery functional coverage, changelog entry, and documentation follow-up.

### Anything else?

- Builds on #2054.
- Partial-parsing behavior is based on dbt-core #9795 / #9844.
- The initial scope would cover SQL query jobs and BigQuery table/partition copy jobs. Python/Dataproc and load jobs would remain out of scope.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.