dbt-labs / dbt-labs/dbt-adapters
[Feature] dbt-athena: allow Python models to materialize the target themselves
- Dominant language
- Python
- Stars
- 233
- Forks
- 362
- Avg merge
- 3d 22h
- Merged PRs (30d)
- 9
Description
### Is this your first time submitting a feature request?
- [x] I have read the [expectations for open source contributors](https://docs.getdbt.com/docs/contributing/oss-expectations)
- [x] I have searched the existing issues, and I could not find an existing issue for this feature
- [x] I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion
### Describe the feature
I'd like a supported way for a Python model's `model()` function to take responsibility for writing the target table itself, so the adapter does not write to it afterwards.
Today the Spark submission template calls `materialize(spark, df, dbt.this)` unconditionally after `model()` runs:
```python
dbt = SparkdbtObj()
df = model(dbt, spark)
materialize(spark, df, dbt.this)
```
There is no supported way to tell the adapter "I already wrote `dbt.this` myself — please don't write again." Returning a DataFrame causes a second write to the same target, and any other return value fails the internal `isinstance` check.
The exact opt-out mechanism is an implementation detail (a sentinel return value, a config flag, a dedicated exception, etc.). The feature I'm asking for is the capability of user-managed writes from Python models on Athena, with a "did the user actually create the target?" guard so silent mistakes still fail loudly.
### Describe alternatives you've considered
Within today's API:
- **Return a DataFrame and accept the double write** — the adapter's write can silently overwrite the user's data, and for Iceberg in particular it can leave the table in an unexpected state.
- **Write to a different target and rename** — extra S3/Glue churn, harder to reason about for Iceberg, and breaks `dbt.this`-based downstream conventions.
- **Use a SQL incremental model with a custom strategy** — does not solve the cases where the merge semantics are only available in Spark Iceberg (see below).
Possible shapes for the opt-out itself (not prescribing one in this issue, just for context):
- Sentinel return value from `model()` (e.g. `return None`) — no new config / class / import, mirrors the existing `return df` convention.
- New config knob (e.g. `skip_python_materialize=True`) — explicit but adds a config surface.
- Dedicated exception (e.g. `raise MaterializedExternally`) — most explicit but requires exporting and importing a class.
### Who will this benefit?
Any Python model on Athena that needs to drive the write itself. The motivating case is Iceberg `MERGE INTO` from a Python model:
- Trino's `MERGE` has a per-writer 100-partition limit, which is restrictive for tables bucketed on a high-cardinality column.
- Spark's Iceberg `MERGE` has no such limit, so users want to run the merge from a Python model via `session.sql("MERGE INTO ...")` and have the adapter stay out of the write path.
Other use cases include custom Python writers for Delta/Hudi on a user-controlled path, or any flow where the user already owns the catalog write and only wants to use Python for the compute.
Example user model under this feature (using `return None` as the opt-out, but the shape is open):
```python
def model(dbt, session):
dbt.config(materialized="incremental", table_type="iceberg")
df = ... # build source
target_fqn = f"{dbt.this.schema}.{dbt.this.identifier}"
if not dbt.is_incremental:
df.writeTo(target_fqn).using("iceberg").createOrReplace()
else:
df.createOrReplaceTempView("src")
session.sql(f"MERGE INTO {target_fqn} t USING src s ON ...")
return None # adapter does not call materialize()
```
### Are you interested in contributing this feature?
Yes — PR #1990
### Anything else?
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.