dbt-labs / dbt-labs/dbt

[Feature] Specify data type of snapshot generated columns

Open
#11,650 2 comments 4 reactions 0 assignees View on GitHub
engine:v1 Refinement snapshots type:feature
Dominant language
Rust
Stars
13.8k
Forks
2.6k
Avg merge
21h 31m
Merged PRs (30d)
56

Description

### Is this your first time submitting a feature request?

- [x] I have read the [expectations for open source contributors](https://docs.getdbt.com/docs/contributing/oss-expectations)
- [x] I have searched the existing issues, and I could not find an existing issue for this feature
- [x] I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

### Describe the feature

Currently in Snowflake, if I create a dbt snapshot, the `dbt_valid_from`, `dbt_valid_to`, and `dbt_updated_at` columns are created as a UTC timestamp but stored as a `TIMESTAMP_NTZ` type (exact SQL below):

```sql
to_timestamp_ntz(convert_timezone('UTC', current_timestamp()))
```

This can cause discrepancies in downstream usage for teams that primarily use `TIMESTAMP_TZ`. For example all `TIMESTAMP_NTZ` columns are evaluated using the session/account timezone (default is `America/Los Angeles`). That means `'2020-01-01 00:00:00'::TIMESTAMP_NTZ` is treated as _greater than_ `'2020-01-01 00:00:00 +0000'::TIMESTAMP_TZ`.

When creating a snapshot, users should have the ability to specify the data type of the dbt-generated columns.

- Snowflake: Choose from `TIMESTAMP_TZ`, `TIMESTAMP_NTZ`, or `TIMESTAMP_LTZ`
- BigQuery: Choose from `TIMESTAMP` or `DATETIME`
- Databricks: Choose from `TIMESTAMP` or `TIMESTAMP_NTZ`
- Redshift: Choose from `TIMESTAMP` or `TIMESTAMPTZ`

### Describe alternatives you've considered

In the meantime, this requires one of four solutions:

- Manually override dbt snapshots to enable the above functionality (not desired)
- Create a view on top of every snapshot to change the data type
- Let each downstream model decide if the data type should be updated
- Let downstream consumers handle it

### Who will this benefit?

Any users that work downstream of snapshots that require consistency in their timestamp data types. That should cover most users of dbt + dashboard users

### Are you interested in contributing this feature?

No

### Anything else?

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.