aws-samples / aws-samples/dbt-glue
Case-sensitive column comparison causes false schema change detection on case-insensitive adapters
- Dominant language
- Python
- Stars
- 147
- Forks
- 96
- Avg merge
- 7h 4m
- Merged PRs (30d)
- 5
Description
### Describe the bug
When using the on_schema_change configuration (e.g., append_new_columns, insert+overwrite), dbt appears to compare column names in a case-sensitive manner. However, Spark (and other warehouses) treat column names as case-insensitive. This means that if a column's casing changes between runs (e.g., HeyThere → heythere), dbt incorrectly detects a schema change and fails to materialize the model — even though the schema is functionally identical.
### Steps To Reproduce
```sql
-- models/my_model.sql
{{ config(materialized='incremental', on_schema_change='append_new_columns') }}
select 1 as HeyThere
```
then reproduce the model with
```sql
select 1 as heythere
```
### Expected behavior
The model should materialize successfully, since HeyThere and heythere refer to the same column in Spark (case-insensitive).
### Screenshots and log output
If applicable, add screenshots or log output to help explain your problem.
### System information
**The output of `dbt --version`:**
```
1.10.19
```
Contributor guide
Research direction
Start by locating the Python implementation of on_schema_change handling and the schema or column comparison entry point; no specific files or tests are named. Reproduce the issue with the provided incremental model and the HeyThere/heythere rename on a case-insensitive adapter, then verify that equivalent casing no longer triggers a schema-change failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, spark
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100