dbt-labs / dbt-labs/dbt-adapters
[Feature] Support explicit `insert_overwrite` for unpartitioned Athena Hive incremental models
- Dominant language
- Python
- Stars
- 233
- Forks
- 362
- Avg merge
- 3d 22h
- Merged PRs (30d)
- 9
Description
### Is this your first time submitting a feature request?
- [x] I have read the expectations for open source contributors
- [x] I have searched the existing issues, and I could not find an existing issue for this feature
- [x] I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion
### Describe the feature
For unpartitioned Hive models, an explicit `insert_overwrite` performs a full-table replacement when `external_location` is unset (the default) and `s3_data_naming` contains `unique` (the default is `schema_table_unique`). Otherwise, dbt preserves the existing `append` behavior and logs an informational message.
dbt-athena currently falls back to `append` for unpartitioned Hive incremental models, even when `insert_overwrite` is explicitly configured:
```sql
{{
config(
materialized='incremental',
incremental_strategy='insert_overwrite',
table_type='hive',
format='parquet'
)
}}
select 1 as id
```
**Instead of replacing the target table, every incremental run appends the complete model result. For this one-row example, running the model N times produces N identical rows, even though `insert_overwrite` is explicitly configured.**
This is counterintuitive from a user's perspective: an explicitly requested overwrite is expected to replace existing data, not accumulate another copy on every run.
| Configuration | Current behavior | Proposed behavior |
| --- | --- | --- |
| Unpartitioned Hive, strategy omitted | `append` | `append` |
| Unpartitioned Hive, explicit `insert_overwrite`, `external_location` unset (default) and `s3_data_naming` contains `unique` (default: `schema_table_unique`) | `append` | Full-table replacement |
| Unpartitioned Hive, explicit `insert_overwrite`, `external_location` set or `s3_data_naming` does not contain `unique` | `append` | `append` with an informational message |
| Partitioned Hive and other strategies | Existing behavior | Unchanged |
dbt-spark already treats an explicitly configured unpartitioned `insert_overwrite` as a full-table replacement ([implementation](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-spark/src/dbt/include/spark/macros/materializations/incremental/strategies.sql#L1-L15), [functional test](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-spark/tests/functional/adapter/incremental_strategies/test_incremental_strategies.py#L64-L83)).
### Describe alternatives you've considered
Users can continue accepting the `append` fallback or use `--full-refresh`, which drops and recreates the target table. Turning unsupported storage configurations into compilation errors was rejected to preserve backward compatibility.
### Who will this benefit?
Athena users who need to replace all data in an unpartitioned Hive table without using full refresh.
### Are you interested in contributing this feature?
Yes. An implementation and functional tests are available in #2082.
### Anything else?
Minimal reproduction:
https://github.com/AlejandroMorgante/dbt-athena-hive-insert-overwrite-repro
Documentation tracking:
https://github.com/dbt-labs/docs.getdbt.com/issues/9643
Verified with:
- dbt-core 1.12.0
- dbt-adapters 1.24.5
- dbt-athena 1.11.0
- Athena engine v3
Contributor guide
Assessment
This issue has not been assessed yet.