dbt-labs / dbt-labs/dbt-adapters
[Feature] Continue BigQuery jobs across dbt invocations
- Dominant language
- Python
- Stars
- 233
- Forks
- 362
- Avg merge
- 3d 22h
- Merged PRs (30d)
- 9
Description
### Is this your first time submitting a feature request?
- [x] I have read the [expectations for open source contributors](https://docs.getdbt.com/docs/contributing/oss-expectations)
- [x] I have searched the existing issues, and I could not find an existing issue for this feature
- [x] I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion
### Describe the feature
Allow a later `dbt-bigquery` invocation to attach to existing BigQuery jobs and continue the same logical orchestrator task.
PR #2054 made query and copy job submission idempotent within one dbt process by retaining a generated job ID across client retries and attaching on `409 Conflict`. A new dbt process still generates new IDs, however. If an orchestrator retries a task on another worker after the original dbt process exits while its BigQuery job continues, the retry can submit duplicate work.
The proposed opt-in behavior is:
- Accept a runtime-only BigQuery profile credential
- Derive stable job IDs for query and copy operations from the logical key plus node, microbatch, and statement/copy identity
- Attach to matching jobs created by a prior dbt invocation
- Validate a request fingerprint before attaching, and fail closed if SQL or semantic job configuration differs
- Return ordinary adapter results so the new invocation can continue dbt's materialization control flow after the previously submitted job
The runtime-only credential should be omitted from `BigQueryCredentials._connection_keys()`. Since dbt-core #9844 hashes only `connection_info()` for partial parsing, this keeps a per-run orchestration value out of `{{ target }}` and avoids invalidating partial parsing.
### Describe alternatives you've considered
- A fixed adapter environment variable is parse-neutral but bypasses typed profile validation and is awkward for programmatic callers.
- A new dbt-core CLI flag would introduce an adapter-specific Core interface; adapters do not currently expose a CLI-extension hook.
### Who will this benefit?
Users running dbt under Airflow, Kubernetes, or another distributed orchestrator where an automatic retry may execute in a fresh process or on a different machine. It prevents duplicate BigQuery work and lets the retry wait for and consume the original job's result.
### Are you interested in contributing this feature?
Yes. I can contribute the implementation, unit coverage, BigQuery functional coverage, changelog entry, and documentation follow-up.
### Anything else?
- Builds on #2054.
- Partial-parsing behavior is based on dbt-core #9795 / #9844.
- The initial scope would cover SQL query jobs and BigQuery table/partition copy jobs. Python/Dataproc and load jobs would remain out of scope.
Contributor guide
Assessment
This issue has not been assessed yet.