dbt-labs / dbt-labs/dbt-adapters

[Bug] dbt-athena: TOO_MANY_OPEN_PARTITIONS crashes with TypeError for unpartitioned models

Open
#1,742 0 comments 0 reactions 0 assignees View on GitHub
triage:product type:bug
Dominant language
Python
Stars
233
Forks
362
Avg merge
3d 22h
Merged PRs (30d)
9

Description

### Is this a new bug?

- [x] I believe this is a new bug
- [x] I have searched the existing issues, and I could not find an existing issue for this bug

### Which packages are affected?

- [ ] dbt-adapters
- [ ] dbt-tests-adapter
- [x] dbt-athena
- [ ] dbt-athena-community
- [ ] dbt-bigquery
- [ ] dbt-postgres
- [ ] dbt-redshift
- [ ] dbt-snowflake
- [ ] dbt-spark

### Current Behavior

When `create_table_as_with_partitions` is called and the source query returns zero rows, `get_partition_batches` returns an empty list. The batch loop is never entered, so the target table is **never created**.

Any post-hook referencing the relation (e.g. `OPTIMIZE` or `VACUUM`) then fails:

```
TABLE_NOT_FOUND: Table ... does not exist
```

`create_table_as_with_partitions` is called in two situations:

- `force_batch=True` is configured on the model
- `safe_create_table_as` falls back to it after a `TOO_MANY_OPEN_PARTITIONS` error

In both cases, if the source data happens to be empty (e.g. an incremental model with no new rows in the current run), the target table is silently not created.

### Expected Behavior

The target table should always be created after `create_table_as_with_partitions` completes, even when the source data is empty. An empty table is a valid result for a zero-row run.

### Steps To Reproduce

1. Configure a partitioned Iceberg model with `force_batch=True` and a post-hook:
```yaml
config:
materialized: incremental
incremental_strategy: insert_overwrite
partitioned_by: [date_col]
force_batch: true
post_hook: "OPTIMIZE {{ this }} REWRITE DATA USING BIN_PACK"
```
2. Run the model when the source query returns **zero rows**.
3. `get_partition_batches` returns `[]`, the batch loop is skipped, and no target table is created.
4. The post-hook runs and fails with `TABLE_NOT_FOUND`.

### Relevant log output

```shell
Compilation Error in model my_model (models/my_model.sql)
'NoneType' object is not iterable
> in macro create_table_as_with_partitions (macros/materializations/models/table/create_table_as.sql)
> called by macro safe_create_table_as (macros/materializations/models/table/create_table_as.sql)
```

### Environment

```markdown
- OS: macOS
- Python: 3.12.8
- dbt-adapters: 1.22.8
- dbt-athena: 1.9.4
```

### Additional Context

Root cause is in `create_table_as_with_partitions` (`macros/materializations/models/table/create_table_as.sql`):

```sql
{% set partitions_batches = get_partition_batches(sql=tmp_relation, as_subquery=False) %}

{%- for batch in partitions_batches -%}
{%- if loop.index == 1 -%}
{# CREATE TABLE AS SELECT ... WHERE batch — only runs if batches exist #}
{%- else -%}
{# INSERT INTO ... WHERE batch #}
{%- endif -%}
{%- endfor -%}

{# If partitions_batches is empty, the loop never runs and no table is created #}

{%- do drop_relation(tmp_relation) -%}
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.