dbt-labs / dbt-labs/dbt-adapters
[Regression] support non-literal `batch_id` config for python models on dataproc
- Dominant language
- Python
- Stars
- 233
- Forks
- 362
- Avg merge
- 3d 22h
- Merged PRs (30d)
- 9
Description
### Is this your first time submitting a feature request?
- [X] I have read the [expectations for open source contributors](https://docs.getdbt.com/docs/contributing/oss-expectations)
- [X] I have searched the existing issues, and I could not find an existing issue for this feature
- [X] I am requesting a straightforward extension of existing dbt-bigquery functionality, rather than a Big Idea better suited to a discussion
### Describe the feature
currently, the default batch ID that's included for python models submitted to dataproc is simply `str(uuid.uuid4())`, this was last changed with dbt-labs/dbt-bigquery#1020.
this works, and is sufficient to avoid `409 Already exists: Failed to create batch` errors from dataproc when attempting to submit batches with duplicate names, but after the test changes included in dbt-labs/dbt-bigquery#1014, attempting to pass any non-literal `batch_id` in the model config will cause a parsing error, e.g.;
```
18:19:35 Running with dbt=1.8.5
18:19:36 Registered adapter: bigquery=1.8.2
18:19:36 Unable to do partial parsing because of a version mismatch
18:19:39 Encountered an error:
Parsing Error
Error when trying to literal_eval an arg to dbt.ref(), dbt.source(), dbt.config() or dbt.config.get()
malformed node or string on line 49:
https://docs.python.org/3/library/ast.html#ast.literal_eval
In dbt python model, `dbt.ref`, `dbt.source`, `dbt.config`, `dbt.config.get` function args only support Python literal structures
```
this makes passing any non-default `batch_id` more or less impossible, as using a var to assign a dynamic batch ID at runtime will throw an error from `literal_eval`, and setting a static batch ID will allow a model to run on dataproc only once before throwing a 409 error.
### Describe alternatives you've considered
one alternative would be to amend the `default_batch_id` config to prepend the model name with either a uuid, or with a non-static dbt env var, maybe `invocation_id` (unsure if this would only work on dbt cloud)? this would avoid the previous errors when using `created_at` as mentioned in dbt-labs/dbt-bigquery#1006
### Who will this benefit?
everyone who wants to see descriptive batch names in dataproc!
### Are you interested in contributing this feature?
yes, I'm a regular dbt user but haven't contributed anything here before :)
### Anything else?
I've confirmed this is broken in both dbt-core v1.8.5/dbt-bigquery v1.8.2 and dbt-core v1.7.16/dbt-bigquery v1.7.9
Contributor guide
Assessment
This issue has not been assessed yet.