dbt-labs / dbt-labs/dbt

[CT-2230] [Bug] snapshots not working properly with hive when specified column is not lowercase

Open
#7,084 5 comments 0 reactions 0 assignees View on GitHub
engine:v1 help_wanted snapshots type:bug
Dominant language
Rust
Stars
13.8k
Forks
2.6k
Avg merge
21h 31m
Merged PRs (30d)
56

Description

### Is this a new bug in dbt-core?

- [X] I believe this is a new bug in dbt-core
- [X] I have searched the existing issues, and I could not find an existing issue for this bug

### Current Behavior

Hive metastore stores all column names as lowercase, even if they're explicitly created with uppercase names.
When you write a snapshot config that has the `check` strategy, and specify columns with uppercase column names, `dbt snapshot` invalidates all old records and copies all new records, even if there are no changes at all.

### Expected Behavior

`dbt snapshot` throws an error if the column does not exist.
For this specific case with hive metastore: if `UPPERCASE_COLUMN` gets stored as `uppercase_column`, but all the snapshot config is still written with `UPPERCASE_COLUMN`, `dbt snapshot` should complain that `UPPERCASE_COLUMN` does not exist.

### Steps To Reproduce

1. Have an adapter that uses hive metastore
2. Create a table with an uppercase column, e.g. `UPPERCASE_COLUMN`
3. `select *` from your table, and observe that the column name is now lowercase, as hive metastore has lowercased it
4. Create a snapshot from this table with the `check` strategy and specifiy the `UPPERCASE_COLUMN` as a column to check
5. Create another snapshot from this table (without changing anything, just run `dbt snapshot` again)
6. Select * from your snapshot and observe that the data is in there twice, with half of the records invalidated

### Relevant log output

_No response_

### Environment

```markdown
- OS: Ubuntu 22.04
- Python: 3.10.6
- dbt: 1.4.1
```

### Which database adapter are you using with dbt?

other (mention it in "Additional Context")

### Additional Context

We use the trino adapter, with a hive metastore and underlying iceberg.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.