dbt-labs / dbt-labs/dbt-adapters
[Bug] Incremental materialization recreates tables when their existence can't be confirmed
- Dominant language
- Python
- Stars
- 233
- Forks
- 362
- Avg merge
- 3d 22h
- Merged PRs (30d)
- 9
Description
### Is this a new bug?
- [x] I believe this is a new bug
- [x] I have searched the existing issues, and I could not find an existing issue for this bug
### Which packages are affected?
- [ ] dbt-adapters
- [ ] dbt-tests-adapter
- [ ] dbt-athena
- [ ] dbt-athena-community
- [ ] dbt-bigquery
- [ ] dbt-postgres
- [ ] dbt-redshift
- [ ] dbt-snowflake
- [x] dbt-spark
### Current Behavior
We run dbt on Spark on Kubernetes with Kyuubi, Delta and Hive Metastore.
Recently, we have had issues with the Metastore, which was restarting due to OOM errors. Consequently, Spark could not connect to it and queries like `show table extended in like '*'` were failing.
What happened next was unexpected. Instead of failing the run, dbt assumed that the table did not exist and executed the `create or replace` query, overwriting all the existing data in the table.
### Expected Behavior
We expect dbt to fail if it cannot accurately determine if a table exists.
Looking through the source code, [I've found that the adapter suppresses errors in `SparkAdapter.list_relations_without_caching` method and returns an empty list](https://github.com/dbt-labs/dbt-adapters/blob/87e81a47baa11c312003377091a9efc0ab72d88e/dbt-spark/src/dbt/adapters/spark/impl.py#L262).
I understand that changing the default behavior might be risky, but this seems like a critical issue. It would be great to at least see this behavior configurable.
For example, we have the following flags enabled in our dbt project configuration to avoid similar kind of silent errors:
```
warn_error_options:
error:
- NoNodesForSelectionCriteria
- NothingToDo
```
Adding something along those lines seems appropriate:
```
warn_error_options:
error:
- NoNodesForSelectionCriteria
- NothingToDo
- UnableToDetermineIfRelationExists
```
### Steps To Reproduce
I'm not sure how to reliably reproduce it, but one could try:
1. Create table with incremental materialization.
2. Run it a few times.
3. Cause your Hive Metastore to fail.
4. See dbt recreate your table.
### Relevant log output
```shell
[2025-08-23, 20:31:34 UTC] {pod_manager.py:535} INFO - [base] /* {"app": "dbt", "dbt_version": "1.9.8", "profile_name": "rivendell", "target_name": "prod", "node_id": "model.rivendell.silver__dictionary__currency_rates"} */
[2025-08-23, 20:31:34 UTC] {pod_manager.py:535} INFO - [base] show table extended in dictionary like '*'
[2025-08-23, 20:31:34 UTC] {pod_manager.py:535} INFO - [base] 20:31:17 Spark adapter: Database Error
[2025-08-23, 20:31:34 UTC] {pod_manager.py:535} INFO - [base] org.apache.kyuubi.KyuubiSQLException: org.apache.kyuubi.KyuubiSQLException: Error operating ExecuteStatement: org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to fetch table cities. java.net.SocketException: Broken pipe (Write failed)
...
2025-08-23, 20:31:34 UTC] {pod_manager.py:535} INFO - [base] 20:31:17 Spark adapter: Error while running:
[2025-08-23, 20:31:34 UTC] {pod_manager.py:535} INFO - [base] macro list_relations_without_caching
[2025-08-23, 20:31:34 UTC] {pod_manager.py:535} INFO - [base] 20:31:17 Spark adapter: Runtime Error
[2025-08-23, 20:31:34 UTC] {pod_manager.py:535} INFO - [base] Database Error
[2025-08-23, 20:31:34 UTC] {pod_manager.py:535} INFO - [base] org.apache.kyuubi.KyuubiSQLException: org.apache.kyuubi.KyuubiSQLException: Error operating ExecuteStatement: org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to fetch table cities. java.net.SocketException: Broken pipe (Write failed)
...
[2025-08-23, 20:31:34 UTC] {pod_manager.py:535} INFO - [base] at java.lang.Thread.run(Thread.java:750)
[2025-08-23, 20:31:34 UTC] {pod_manager.py:535} INFO - [base] 20:31:17 Spark adapter: Error while retrieving information about dictionary: Runtime Error
[2025-08-23, 20:31:34 UTC] {pod_manager.py:535} INFO - [base] Database Error
[2025-08-23, 20:31:34 UTC] {pod_manager.py:535} INFO - [base] org.apache.kyuubi.KyuubiSQLException: org.apache.kyuubi.KyuubiSQLException: Error operating ExecuteStatement: org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to fetch table cities. java.net.SocketException: Broken pipe (Write failed)
[2025-08-23, 20:31:34 UTC] {pod_manager.py:535} INFO - [base] at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:110)
[2025-08-23, 20:31:34 UTC] {pod_manager.py:535} INFO - [base] at org.apache.spark.sql.hive.HiveExternalCatalog.getTable(HiveExternalCatalog.scala:729)
...
{pod_manager.py:535} INFO - [base] 20:31:17 While listing relations in database=, schema=dictionary, found:
[2025-08-23, 20:31:34 UTC] {pod_manager.py:535} INFO - [base] 20:31:17 Using spark connection "model.rivendell.silver__dictionary__currency_rates"
[2025-08-23, 20:31:34 UTC] {pod_manager.py:535} INFO - [base] 20:31:17 On model.rivendell.silver__dictionary__currency_rates: /* {"app": "dbt", "dbt_version": "1.9.8", "profile_name": "rivendell", "target_name": "prod", "node_id": "model.rivendell.silver__dictionary__currency_rates"} */
[2025-08-23, 20:31:34 UTC] {pod_manager.py:535} INFO - [base] msck repair table bronze.ark_db__currency_rates_log;
[2025-08-23, 20:31:34 UTC] {pod_manager.py:535} INFO - [base] 20:31:19 Spark adapter: Poll status: 2, query complete
[2025-08-23, 20:31:34 UTC] {pod_manager.py:535} INFO - [base] 20:31:19 SQL status: OK in 2.564 seconds
[2025-08-23, 20:31:34 UTC] {pod_manager.py:535} INFO - [base] 20:31:19 Writing runtime sql for node "model.rivendell.silver__dictionary__currency_rates"
[2025-08-23, 20:31:34 UTC] {pod_manager.py:535} INFO - [base] 20:31:19 Using spark connection "model.rivendell.silver__dictionary__currency_rates"
[2025-08-23, 20:31:34 UTC] {pod_manager.py:535} INFO - [base] 20:31:19 On model.rivendell.silver__dictionary__currency_rates: /* {"app": "dbt", "dbt_version": "1.9.8", "profile_name": "rivendell", "target_name": "prod", "node_id": "model.rivendell.silver__dictionary__currency_rates"} */
[2025-08-23, 20:31:34 UTC] {pod_manager.py:535} INFO - [base] create or replace table dictionary.currency_rates
[2025-08-23, 20:31:34 UTC] {pod_manager.py:535} INFO - [base] using delta
```
### Environment
```markdown
- OS:
- Python: 3.11
- dbt-adapters: 1.16.3
- dbt-spark: 1.9.8
```
### Additional Context
Model config:
```
{{
config(
alias="currency_rates",
incremental_strategy="merge",
unique_key=["pdate", "src", "dst"],
pre_hook=[
"""
msck repair table {{ source('bronze__airbyte__ark_db__currency_rates', 'ark_db__currency_rates_log') }};
"""
],
post_hook=["{{ optimize(zorder_by='pdate, src, dst') }}"],
)
}}
```
Contributor guide
Assessment
This issue has not been assessed yet.