dbt-labs / dbt-labs/dbt-adapters
[Feature] Support BigQuery Vector Search in dbt-bigquery
- Dominant language
- Python
- Stars
- 233
- Forks
- 362
- Avg merge
- 3d 22h
- Merged PRs (30d)
- 9
Description
### Is this your first time submitting a feature request?
- [x] I have read the [expectations for open source contributors](https://docs.getdbt.com/docs/contributing/oss-expectations)
- [x] I have searched the existing issues, and I could not find an existing issue for this feature
- [x] I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion
### Describe the feature
I'd like dbt-bigquery to natively support the lifecycle management of BigQuery Vector Indexes. This involves:
Defining vector indexes in model configurations (YAML or `config()` macro).
Automatically generating `CREATE VECTOR INDEX` and `DROP VECTOR INDEX` statements.
Implementing state-aware management via the "Relation Config Lifecycle" pattern to ensure indexes are only updated when the configuration changes.
With the rise of RAG and similarity search, this ensures that embedding-based search indexes are treated as first-class citizens alongside the data models they optimize.
### Describe alternatives you've considered
- Post-hooks: Currently, users must manually manage DDL in post_hook. This lacks state awareness (dbt doesn't know if the index exists or needs an update) and is error-prone.
- Dedicated Materialization: Rejected in favor of model-level configuration, which is consistent with existing BigQuery features like clustering and search_index.
### Who will this benefit?
Data engineers and ML engineers building AI applications on BigQuery. This simplifies the deployment of search-optimized tables for semantic search, recommendation engines, and customer support bots.
### Are you interested in contributing this feature?
Yes
### Anything else?
### Implementation Idea:
I plan to implement BigQueryVectorIndexConfig using the dbt-adapters standard. This will support:
- IVF and TREE_AH algorithms.
- STORING clause for pre-filtering optimizations.
- Automatic derivation of PARTITION BY for partitioned tables.
**Example Configuration:schema.yml:**
```
models:
- name: my_vector_table
config:
vector_indexes:
- name: my_v_index
column: embedding
index_type: IVF
distance_type: COSINE
storing: [id, metadata]
ivf_options: {"num_lists": 100}
```
**model_sql.sql:**
```
{{ config(
materialized='table',
partition_by={"field": "created_at", "data_type": "timestamp"},
vector_indexes=[{"column": "embedding", "index_type": "TREE_AH"}]
) }}
select ...
```
#### References:
[BigQuery Vector Search DDL Documentation](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#create_vector_index_statement)
Contributor guide
Assessment
This issue has not been assessed yet.