BigQuery table metadata causing Plan Permadiff
- Dominant language
- Python
- Stars
- 7.3k
- Forks
- 1.4k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 15
Description
## Expected Behavior
Metadata diffs such as `created_timestamp` and `last_updated_timestamp` shouldn't be included in the output of `feast plan` for BigQuery tables created outside of Feast (e.g. with `table_create_disposition` set to `CREATE_NEVER`). This prevents permadiffs in the plan.
## Current Behavior
Feast plan currently creates a permadiff when BigQuery tables are created outside of Feast. This diff is never executed because of the `table_create_disposition` being set to `CREATE_NEVER` thus it is clutter and not relevant as there is no way this plan is every 'actioned'. This is challenging for us as we write out plans to Slack channels and it creates an enormous amount of clutter.
```
Updated feature view [1m[33mmy_feature_view[0m
batch_source: [1m[33mtype: BATCH_BIGQUERY
timestamp_field: "feature_timestamp"
bigquery_options {
table: "my_feature_view"
}
data_source_class_type: "feast.infra.offline_stores.bigquery_source.BigQuerySource"
name: "my_feature_view"
meta {
created_timestamp {
seconds: 1755626119
nanos: 37380000
}
last_updated_timestamp {
seconds: 1755626140
nanos: 128836000
}
}
[0m -> [1m[92mtype: BATCH_BIGQUERY
timestamp_field: "feature_timestamp"
bigquery_options {
table: "my_feature_view"
}
data_source_class_type: "feast.infra.offline_stores.bigquery_source.BigQuerySource"
name: "my_feature_view"
meta {
created_timestamp {
seconds: 1755632676
nanos: 249135000
}
last_updated_timestamp {
seconds: 1755632676
nanos: 249135000
}
}
```
## Steps to reproduce
1. Create BigQuery table outside of Feast.
2. Set `create_table_disposition` to `CREATE_NEVER` in YAML config.
3. Add table to Feast registry with code similar to:
```
my_feature_view_source = BigQuerySource(
name="my_feature_view_batch",
table="my_feature_view_batch",
timestamp_field="feature_timestamp",
)
my_feature_view_push = PushSource(
name="my_feature_view",
batch_source=my_feature_view,
)
```
4. Run `feast plan` and `feast apply` using some principal without access to change anything in BigQuery.
5. Run another `feast plan` and observe the proposed metadata changes that can never be actioned.
### Specifications
- Version: `feast[gcp,aws,grpcio,postgres]==0.51.0`
- Platform: Debian, Py 3.12
- Subsystem: N/a
## Possible Solution
* Exclude `created_at` from the plan as this is immutable.
* Where `create_table_disposition` is set to `CREATE_NEVER`, plans should fail when changes are required to the table.
* `last_updated_timestamp` should never be included in the plan when `create_table_disposition` is set to `CREATE_NEVER` as no update to the table is taking place.
Contributor guide
Assessment
This issue has not been assessed yet.