feast-dev / feast-dev/feast

BigQuery table metadata causing Plan Permadiff

Open
#5,573 0 comments 2 reactions 0 assignees View on GitHub
kind/bug priority/p2
Dominant language
Python
Stars
7.3k
Forks
1.4k
Avg merge
1d 21h
Merged PRs (30d)
15

Description

## Expected Behavior
Metadata diffs such as `created_timestamp` and `last_updated_timestamp` shouldn't be included in the output of `feast plan` for BigQuery tables created outside of Feast (e.g. with `table_create_disposition` set to `CREATE_NEVER`). This prevents permadiffs in the plan.

## Current Behavior
Feast plan currently creates a permadiff when BigQuery tables are created outside of Feast. This diff is never executed because of the `table_create_disposition` being set to `CREATE_NEVER` thus it is clutter and not relevant as there is no way this plan is every 'actioned'. This is challenging for us as we write out plans to Slack channels and it creates an enormous amount of clutter.

```
Updated feature view [1m[33mmy_feature_view[0m
batch_source: [1m[33mtype: BATCH_BIGQUERY
timestamp_field: "feature_timestamp"
bigquery_options {
table: "my_feature_view"
}
data_source_class_type: "feast.infra.offline_stores.bigquery_source.BigQuerySource"
name: "my_feature_view"
meta {
created_timestamp {
seconds: 1755626119
nanos: 37380000
}
last_updated_timestamp {
seconds: 1755626140
nanos: 128836000
}
}
[0m -> [1m[92mtype: BATCH_BIGQUERY
timestamp_field: "feature_timestamp"
bigquery_options {
table: "my_feature_view"
}
data_source_class_type: "feast.infra.offline_stores.bigquery_source.BigQuerySource"
name: "my_feature_view"
meta {
created_timestamp {
seconds: 1755632676
nanos: 249135000
}
last_updated_timestamp {
seconds: 1755632676
nanos: 249135000
}
}
```

## Steps to reproduce

1. Create BigQuery table outside of Feast.
2. Set `create_table_disposition` to `CREATE_NEVER` in YAML config.
3. Add table to Feast registry with code similar to:
```
my_feature_view_source = BigQuerySource(
name="my_feature_view_batch",
table="my_feature_view_batch",
timestamp_field="feature_timestamp",
)

my_feature_view_push = PushSource(
name="my_feature_view",
batch_source=my_feature_view,
)
```
4. Run `feast plan` and `feast apply` using some principal without access to change anything in BigQuery.
5. Run another `feast plan` and observe the proposed metadata changes that can never be actioned.

### Specifications

- Version: `feast[gcp,aws,grpcio,postgres]==0.51.0`
- Platform: Debian, Py 3.12
- Subsystem: N/a

## Possible Solution
* Exclude `created_at` from the plan as this is immutable.
* Where `create_table_disposition` is set to `CREATE_NEVER`, plans should fail when changes are required to the table.
* `last_updated_timestamp` should never be included in the plan when `create_table_disposition` is set to `CREATE_NEVER` as no update to the table is taking place.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.