dbt-labs / dbt-labs/dbt-adapters

[Feature] dbt-athena: allow Python models to materialize the target themselves

Open
#2,002 0 comments 0 reactions 0 assignees View on GitHub
triage:product type:enhancement
Dominant language
Python
Stars
233
Forks
362
Avg merge
3d 22h
Merged PRs (30d)
9

Description

### Is this your first time submitting a feature request?

- [x] I have read the [expectations for open source contributors](https://docs.getdbt.com/docs/contributing/oss-expectations)
- [x] I have searched the existing issues, and I could not find an existing issue for this feature
- [x] I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

### Describe the feature

I'd like a supported way for a Python model's `model()` function to take responsibility for writing the target table itself, so the adapter does not write to it afterwards.

Today the Spark submission template calls `materialize(spark, df, dbt.this)` unconditionally after `model()` runs:

```python
dbt = SparkdbtObj()
df = model(dbt, spark)
materialize(spark, df, dbt.this)
```

There is no supported way to tell the adapter "I already wrote `dbt.this` myself — please don't write again." Returning a DataFrame causes a second write to the same target, and any other return value fails the internal `isinstance` check.

The exact opt-out mechanism is an implementation detail (a sentinel return value, a config flag, a dedicated exception, etc.). The feature I'm asking for is the capability of user-managed writes from Python models on Athena, with a "did the user actually create the target?" guard so silent mistakes still fail loudly.

### Describe alternatives you've considered

Within today's API:

- **Return a DataFrame and accept the double write** — the adapter's write can silently overwrite the user's data, and for Iceberg in particular it can leave the table in an unexpected state.
- **Write to a different target and rename** — extra S3/Glue churn, harder to reason about for Iceberg, and breaks `dbt.this`-based downstream conventions.
- **Use a SQL incremental model with a custom strategy** — does not solve the cases where the merge semantics are only available in Spark Iceberg (see below).

Possible shapes for the opt-out itself (not prescribing one in this issue, just for context):

- Sentinel return value from `model()` (e.g. `return None`) — no new config / class / import, mirrors the existing `return df` convention.
- New config knob (e.g. `skip_python_materialize=True`) — explicit but adds a config surface.
- Dedicated exception (e.g. `raise MaterializedExternally`) — most explicit but requires exporting and importing a class.

### Who will this benefit?

Any Python model on Athena that needs to drive the write itself. The motivating case is Iceberg `MERGE INTO` from a Python model:

- Trino's `MERGE` has a per-writer 100-partition limit, which is restrictive for tables bucketed on a high-cardinality column.
- Spark's Iceberg `MERGE` has no such limit, so users want to run the merge from a Python model via `session.sql("MERGE INTO ...")` and have the adapter stay out of the write path.

Other use cases include custom Python writers for Delta/Hudi on a user-controlled path, or any flow where the user already owns the catalog write and only wants to use Python for the compute.

Example user model under this feature (using `return None` as the opt-out, but the shape is open):

```python
def model(dbt, session):
dbt.config(materialized="incremental", table_type="iceberg")
df = ... # build source
target_fqn = f"{dbt.this.schema}.{dbt.this.identifier}"
if not dbt.is_incremental:
df.writeTo(target_fqn).using("iceberg").createOrReplace()
else:
df.createOrReplaceTempView("src")
session.sql(f"MERGE INTO {target_fqn} t USING src s ON ...")
return None # adapter does not call materialize()
```

### Are you interested in contributing this feature?

Yes — PR #1990

### Anything else?

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.