aws-samples / aws-samples/dbt-glue

Case-sensitive column comparison causes false schema change detection on case-insensitive adapters

Open
#673 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
147
Forks
96
Avg merge
7h 4m
Merged PRs (30d)
5

Description

### Describe the bug
When using the on_schema_change configuration (e.g., append_new_columns, insert+overwrite), dbt appears to compare column names in a case-sensitive manner. However, Spark (and other warehouses) treat column names as case-insensitive. This means that if a column's casing changes between runs (e.g., HeyThere → heythere), dbt incorrectly detects a schema change and fails to materialize the model — even though the schema is functionally identical.

### Steps To Reproduce
```sql
-- models/my_model.sql
{{ config(materialized='incremental', on_schema_change='append_new_columns') }}

select 1 as HeyThere
```
then reproduce the model with
```sql
select 1 as heythere
```

### Expected behavior
The model should materialize successfully, since HeyThere and heythere refer to the same column in Spark (case-insensitive).

### Screenshots and log output
If applicable, add screenshots or log output to help explain your problem.

### System information
**The output of `dbt --version`:**
```
1.10.19
```

Contributor guide

Open the contributing guide

Research direction

Start by locating the Python implementation of on_schema_change handling and the schema or column comparison entry point; no specific files or tests are named. Reproduce the issue with the provided incremental model and the HeyThere/heythere rename on a case-insensitive adapter, then verify that equivalent casing no longer triggers a schema-change failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, spark
Domain
data-engineering, databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.