dbt-labs / dbt-labs/dbt-adapters
[Bug] Contract validation fails for TIMESTAMP WITH TIME ZONE when force_batch triggers fallback mechanism
- Dominant language
- Python
- Stars
- 233
- Forks
- 362
- Avg merge
- 3d 22h
- Merged PRs (30d)
- 9
Description
### Is this a new bug?
- [x] I believe this is a new bug
- [x] I have searched the existing issues, and I could not find an existing issue for this bug
### Which packages are affected?
- [ ] dbt-adapters
- [ ] dbt-tests-adapter
- [x] dbt-athena
- [ ] dbt-athena-community
- [ ] dbt-bigquery
- [ ] dbt-postgres
- [ ] dbt-redshift
- [ ] dbt-snowflake
- [ ] dbt-spark
### Current Behavior
Contract validation fails for `TIMESTAMP WITH TIME ZONE` columns when the `force_batch` configuration or `TOO_MANY_OPEN_PARTITIONS` error triggers the fallback mechanism in the dbt-athena adapter. The same model with identical SQL passes contract validation when using the direct table creation path, but fails when the fallback batch process is used.
**Error Message:**
```
This model has an enforced contract that failed.
Please ensure the name, data_type, and number of columns in your contract match the columns in your model's definition.
| column_name | definition_type | contract_type | mismatch_reason |
| ----------- | --------------- | ------------------------ | ------------------ |
| ts_from_iso | TIMESTAMP | TIMESTAMP WITH TIME ZONE | data type mismatch |
```
The contract validation reports `TIMESTAMP` (actual) vs `TIMESTAMP WITH TIME ZONE` (expected) for columns that are explicitly cast to `TIMESTAMP WITH TIME ZONE` in the SQL.
### Expected Behavior
Contract validation should consistently pass or fail regardless of whether the direct table creation or fallback batch mechanism is used, as long as the SQL logic is identical. If the SQL explicitly casts columns to `TIMESTAMP WITH TIME ZONE`, the contract validation should recognize this consistently across both execution paths.
### Steps To Reproduce
1. **Create a test model with TIMESTAMP WITH TIME ZONE columns:**
```sql
-- prep_data_dbt_contract_test_model.sql
with raw_data_generator as (
select 1 as event_id, '2023-10-26T10:00:00.123456Z' as iso_timestamp_str
union all
select 2 as event_id, '2023-11-01T22:30:00.654321+01:00' as iso_timestamp_str
)
select
event_id,
cast(from_iso8601_timestamp(iso_timestamp_str) as timestamp(6) with time zone) as ts_from_iso
from raw_data_generator
```
2. **Configure with contract enforcement (working case):**
```yaml
# properties.yml
models:
- name: prep_data_dbt_contract_test_model
config:
materialized: table
table_type: iceberg
partitioned_by: ["event_id"]
contract:
enforced: true
# force_batch: false # Default - uses direct creation
columns:
- name: event_id
data_type: integer
- name: ts_from_iso
data_type: timestamp with time zone
```
3. **Run the model (should succeed):**
```bash
dbt run --select prep_data_dbt_contract_test_model --full-refresh
```
✅ **Result: SUCCESS**
4. **Change configuration to force fallback mechanism:**
```yaml
# properties.yml - only change this line
force_batch: true # Forces fallback mechanism
```
5. **Run the model again with identical SQL:**
```bash
dbt run --select prep_data_dbt_contract_test_model --full-refresh
```
❌ **Result: CONTRACT FAILURE**
### Relevant log output
```shell
**Working case (force_batch: false):**
[OK created sql table model prod_general_prepared.prep_data_dbt_contract_test_model [OK 2 in 12.47s]]
**Failing case (force_batch: true):**
Compilation Error in model prep_data_dbt_contract_test_model
This model has an enforced contract that failed.
Please ensure the name, data_type, and number of columns in your contract match the columns in your model's definition.
| column_name | definition_type | contract_type | mismatch_reason |
| ----------- | --------------- | ------------------------ | ------------------ |
| ts_from_iso | TIMESTAMP | TIMESTAMP WITH TIME ZONE | data type mismatch |
```
### Environment
```markdown
- OS: macOS
- Python: 3.12
- dbt-core: 1.10.4
- dbt-athena: 1.8.4
```
### Additional Context
### Root Cause Analysis
The issue occurs in the `create_table_as_with_partitions` macro when the fallback mechanism is triggered. The problem is in line 168 of the macro:
```sql
{%- set dest_columns = adapter.get_columns_in_relation(tmp_relation) -%
{%- set dest_cols_csv = dest_columns | map(attribute='quoted') | join(', ') -%}
```
This line extracts only column **names** but loses the original casting logic:
- **Original SQL**: `cast(from_iso8601_timestamp(...) as timestamp(6) with time zone) as ts_from_iso`
- **Fallback SQL**: `SELECT ts_from_iso FROM temp_table` (no explicit cast)
When Athena creates the final table without explicit casts, it infers `TIMESTAMP` instead of `TIMESTAMP WITH TIME ZONE`.
### File References
- **Primary issue location**: [`create_table_as.sql`](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-athena/src/dbt/include/athena/macros/materializations/models/table/create_table_as.sql#L149-L201) (lines 151-201)
- **Contract validation**: [`columns_spec_ddl.sql`](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-adapters/src/dbt/include/global_project/macros/relations/column/columns_spec_ddl.sql#L27-L29) (lines 27-30)
### Production Impact
This affects any production model that:
- Uses `TIMESTAMP WITH TIME ZONE` columns
- Has heavy partitioning that triggers `TOO_MANY_OPEN_PARTITIONS`
- Has contract enforcement enabled
The workaround is to either:
1. Reduce partitioning granularity to avoid triggering the fallback
2. Disable contract enforcement (`contract: enforced: false`)
3. Use `force_batch: false` explicitly (though this may still fail with `TOO_MANY_OPEN_PARTITIONS`)
### Related Configuration
The `force_batch: true` configuration provides a reliable way to reproduce this issue without needing large datasets or complex partitioning schemes, making it an excellent test case for validating any fix.
Contributor guide
Assessment
This issue has not been assessed yet.