dbt-labs / dbt-labs/dbt-adapters
[Bug] dbt-athena: TOO_MANY_OPEN_PARTITIONS crashes with TypeError for unpartitioned models
- Dominant language
- Python
- Stars
- 233
- Forks
- 362
- Avg merge
- 3d 22h
- Merged PRs (30d)
- 9
Description
### Is this a new bug?
- [x] I believe this is a new bug
- [x] I have searched the existing issues, and I could not find an existing issue for this bug
### Which packages are affected?
- [ ] dbt-adapters
- [ ] dbt-tests-adapter
- [x] dbt-athena
- [ ] dbt-athena-community
- [ ] dbt-bigquery
- [ ] dbt-postgres
- [ ] dbt-redshift
- [ ] dbt-snowflake
- [ ] dbt-spark
### Current Behavior
When `create_table_as_with_partitions` is called and the source query returns zero rows, `get_partition_batches` returns an empty list. The batch loop is never entered, so the target table is **never created**.
Any post-hook referencing the relation (e.g. `OPTIMIZE` or `VACUUM`) then fails:
```
TABLE_NOT_FOUND: Table ... does not exist
```
`create_table_as_with_partitions` is called in two situations:
- `force_batch=True` is configured on the model
- `safe_create_table_as` falls back to it after a `TOO_MANY_OPEN_PARTITIONS` error
In both cases, if the source data happens to be empty (e.g. an incremental model with no new rows in the current run), the target table is silently not created.
### Expected Behavior
The target table should always be created after `create_table_as_with_partitions` completes, even when the source data is empty. An empty table is a valid result for a zero-row run.
### Steps To Reproduce
1. Configure a partitioned Iceberg model with `force_batch=True` and a post-hook:
```yaml
config:
materialized: incremental
incremental_strategy: insert_overwrite
partitioned_by: [date_col]
force_batch: true
post_hook: "OPTIMIZE {{ this }} REWRITE DATA USING BIN_PACK"
```
2. Run the model when the source query returns **zero rows**.
3. `get_partition_batches` returns `[]`, the batch loop is skipped, and no target table is created.
4. The post-hook runs and fails with `TABLE_NOT_FOUND`.
### Relevant log output
```shell
Compilation Error in model my_model (models/my_model.sql)
'NoneType' object is not iterable
> in macro create_table_as_with_partitions (macros/materializations/models/table/create_table_as.sql)
> called by macro safe_create_table_as (macros/materializations/models/table/create_table_as.sql)
```
### Environment
```markdown
- OS: macOS
- Python: 3.12.8
- dbt-adapters: 1.22.8
- dbt-athena: 1.9.4
```
### Additional Context
Root cause is in `create_table_as_with_partitions` (`macros/materializations/models/table/create_table_as.sql`):
```sql
{% set partitions_batches = get_partition_batches(sql=tmp_relation, as_subquery=False) %}
{%- for batch in partitions_batches -%}
{%- if loop.index == 1 -%}
{# CREATE TABLE AS SELECT ... WHERE batch — only runs if batches exist #}
{%- else -%}
{# INSERT INTO ... WHERE batch #}
{%- endif -%}
{%- endfor -%}
{# If partitions_batches is empty, the loop never runs and no table is created #}
{%- do drop_relation(tmp_relation) -%}
```
Contributor guide
Assessment
This issue has not been assessed yet.