dbt-labs / dbt-labs/dbt-adapters

[Feature] Support explicit `insert_overwrite` for unpartitioned Athena Hive incremental models

Open
#2,081 0 comments 1 reaction 0 assignees View on GitHub
triage:product
Dominant language
Python
Stars
233
Forks
362
Avg merge
3d 22h
Merged PRs (30d)
9

Description

### Is this your first time submitting a feature request?

- [x] I have read the expectations for open source contributors
- [x] I have searched the existing issues, and I could not find an existing issue for this feature
- [x] I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

### Describe the feature

For unpartitioned Hive models, an explicit `insert_overwrite` performs a full-table replacement when `external_location` is unset (the default) and `s3_data_naming` contains `unique` (the default is `schema_table_unique`). Otherwise, dbt preserves the existing `append` behavior and logs an informational message.

dbt-athena currently falls back to `append` for unpartitioned Hive incremental models, even when `insert_overwrite` is explicitly configured:

```sql
{{
config(
materialized='incremental',
incremental_strategy='insert_overwrite',
table_type='hive',
format='parquet'
)
}}

select 1 as id
```

**Instead of replacing the target table, every incremental run appends the complete model result. For this one-row example, running the model N times produces N identical rows, even though `insert_overwrite` is explicitly configured.**

This is counterintuitive from a user's perspective: an explicitly requested overwrite is expected to replace existing data, not accumulate another copy on every run.

| Configuration | Current behavior | Proposed behavior |
| --- | --- | --- |
| Unpartitioned Hive, strategy omitted | `append` | `append` |
| Unpartitioned Hive, explicit `insert_overwrite`, `external_location` unset (default) and `s3_data_naming` contains `unique` (default: `schema_table_unique`) | `append` | Full-table replacement |
| Unpartitioned Hive, explicit `insert_overwrite`, `external_location` set or `s3_data_naming` does not contain `unique` | `append` | `append` with an informational message |
| Partitioned Hive and other strategies | Existing behavior | Unchanged |

dbt-spark already treats an explicitly configured unpartitioned `insert_overwrite` as a full-table replacement ([implementation](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-spark/src/dbt/include/spark/macros/materializations/incremental/strategies.sql#L1-L15), [functional test](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-spark/tests/functional/adapter/incremental_strategies/test_incremental_strategies.py#L64-L83)).

### Describe alternatives you've considered

Users can continue accepting the `append` fallback or use `--full-refresh`, which drops and recreates the target table. Turning unsupported storage configurations into compilation errors was rejected to preserve backward compatibility.

### Who will this benefit?

Athena users who need to replace all data in an unpartitioned Hive table without using full refresh.

### Are you interested in contributing this feature?

Yes. An implementation and functional tests are available in #2082.

### Anything else?

Minimal reproduction:

https://github.com/AlejandroMorgante/dbt-athena-hive-insert-overwrite-repro

Documentation tracking:

https://github.com/dbt-labs/docs.getdbt.com/issues/9643

Verified with:

- dbt-core 1.12.0
- dbt-adapters 1.24.5
- dbt-athena 1.11.0
- Athena engine v3

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.