dbt-labs / dbt-labs/dbt-adapters

[Feature] Support BigQuery Vector Search in dbt-bigquery

Open
#1,519 1 comment 0 reactions 0 assignees View on GitHub
triage:product type:enhancement
Dominant language
Python
Stars
233
Forks
362
Avg merge
3d 22h
Merged PRs (30d)
9

Description

### Is this your first time submitting a feature request?

- [x] I have read the [expectations for open source contributors](https://docs.getdbt.com/docs/contributing/oss-expectations)
- [x] I have searched the existing issues, and I could not find an existing issue for this feature
- [x] I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

### Describe the feature

I'd like dbt-bigquery to natively support the lifecycle management of BigQuery Vector Indexes. This involves:

Defining vector indexes in model configurations (YAML or `config()` macro).
Automatically generating `CREATE VECTOR INDEX` and `DROP VECTOR INDEX` statements.
Implementing state-aware management via the "Relation Config Lifecycle" pattern to ensure indexes are only updated when the configuration changes.
With the rise of RAG and similarity search, this ensures that embedding-based search indexes are treated as first-class citizens alongside the data models they optimize.

### Describe alternatives you've considered

- Post-hooks: Currently, users must manually manage DDL in post_hook. This lacks state awareness (dbt doesn't know if the index exists or needs an update) and is error-prone.
- Dedicated Materialization: Rejected in favor of model-level configuration, which is consistent with existing BigQuery features like clustering and search_index.

### Who will this benefit?

Data engineers and ML engineers building AI applications on BigQuery. This simplifies the deployment of search-optimized tables for semantic search, recommendation engines, and customer support bots.

### Are you interested in contributing this feature?

Yes

### Anything else?

### Implementation Idea:

I plan to implement BigQueryVectorIndexConfig using the dbt-adapters standard. This will support:

- IVF and TREE_AH algorithms.
- STORING clause for pre-filtering optimizations.
- Automatic derivation of PARTITION BY for partitioned tables.

**Example Configuration:schema.yml:**

```
models:
- name: my_vector_table
config:
vector_indexes:
- name: my_v_index
column: embedding
index_type: IVF
distance_type: COSINE
storing: [id, metadata]
ivf_options: {"num_lists": 100}

```

**model_sql.sql:**

```
{{ config(
materialized='table',
partition_by={"field": "created_at", "data_type": "timestamp"},
vector_indexes=[{"column": "embedding", "index_type": "TREE_AH"}]
) }}
select ...
```

#### References:
[BigQuery Vector Search DDL Documentation](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#create_vector_index_statement)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.