dbt-labs / dbt-labs/dbt-adapters

[Feature] Replacing a view is not supported by catalog: glue_catalog

Open
#1,200 1 comment 0 reactions 0 assignees View on GitHub
pkg:dbt-spark type:enhancement
Dominant language
Python
Stars
233
Forks
362
Avg merge
3d 22h
Merged PRs (30d)
9

Description

### Is this a new bug?

- [x] I believe this is a new bug
- [x] I have searched the existing issues, and I could not find an existing issue for this bug

### Which packages are affected?

- [ ] dbt-adapters
- [ ] dbt-tests-adapter
- [ ] dbt-athena
- [ ] dbt-athena-community
- [ ] dbt-bigquery
- [ ] dbt-postgres
- [ ] dbt-redshift
- [ ] dbt-snowflake
- [x] dbt-spark

### Current Behavior

- Running dbt run --select stg_orders fails with error: java.lang.UnsupportedOperationException: Replacing a view is not supported by catalog: glue_catalog
- The error occurs when dbt attempts to execute CREATE OR REPLACE VIEW against the Glue catalog
- Model is configured with materialized = 'view' in the dbt configuration

### Expected Behavior

- dbt should successfully create/update view models when using Glue catalog
- Either support CREATE OR REPLACE VIEW or implement alternative logic to drop and recreate views

### Steps To Reproduce

## Prerequisites
• AWS environment with Glue catalog configured
• Spark cluster with Iceberg support
• dbt-spark adapter version 1.9.2
• dbt version 1.9.8
• Access to AWS Glue Data Catalog

### Environment Setup
1. Configure dbt profile with Spark connection using Glue catalog:

yaml
# profiles.yml
target: spark_dev
outputs:
spark_dev:
type: spark
method: thrift
host:
port:
# Glue catalog configuration

2. Ensure Glue catalog is using Iceberg format (v2 tables)

### Reproduction Steps

1. Create a simple dbt model with view materialization:

sql
-- models/staging/stg_orders.sql
{{
config(
materialized = 'view'
)
}}

SELECT
order_id,
customer_id,
order_date,
order_amount
FROM {{ source('source_database', 'orders') }}

2. Run the model:
bash
dbt run --select stg_orders

### Relevant log output

```shell
16:26:00 On master: ROLLBACK
16:26:00 Opening a new connection, currently in state init
16:26:00 Spark adapter: NotImplemented: rollback
16:26:00 Spark adapter: NotImplemented: add_begin_query
16:26:00 Spark adapter: NotImplemented: commit
16:26:00 On master: ROLLBACK
16:26:00 Spark adapter: NotImplemented: rollback
16:26:00 On master: Close
16:26:00 Connection 'master' was properly closed.
16:26:00 Connection 'model.aws_dbt_control_framework.stg_orders' was properly closed.
16:26:00
16:26:00 Finished running 1 view model in 0 hours 0 minutes and 7.40 seconds (7.40s).
16:26:00 Command end result
16:26:00 Wrote artifact WritableManifest to /home/ssm-user/aws-dbt-control-framework/target/manifest.json
16:26:00 Wrote artifact SemanticManifest to /home/ssm-user/aws-dbt-control-framework/target/semantic_manifest.json
16:26:00 Wrote artifact RunExecutionResult to /home/ssm-user/aws-dbt-control-framework/target/run_results.json
16:26:00
16:26:00 Completed with 1 error, 0 partial successes, and 0 warnings:
16:26:00
16:26:00 Runtime Error in model stg_orders (models/staging/stg_orders.sql)
Database Error
org.apache.hive.service.cli.HiveSQLException: Error running query: java.lang.UnsupportedOperationException: Replacing a view is not supported by catalog: glue_catalog
at org.apache.spark.sql.hive.thriftserver.HiveThriftServerErrors$.runningQueryError(HiveThriftServerErrors.scala:46)
at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.org$apache$spark$sql$hive$thriftserver$SparkExecuteStatementOperation$$execute(SparkExecuteStatementOperation.scala:263)
at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3.$anonfun$run$2(SparkExecuteStatementOperation.scala:167)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at org.apache.spark.sql.hive.thriftserver.SparkOperation.withLocalProperties(SparkOperation.scala:79)
at org.apache.spark.sql.hive.thriftserver.SparkOperation.withLocalProperties$(SparkOperation.scala:63)
at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.withLocalProperties(SparkExecuteStatementOperation.scala:41)
at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3.run(SparkExecuteStatementOperation.scala:167)
at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3.run(SparkExecuteStatementOperation.scala:162)
at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
at java.base/javax.security.auth.Subject.doAs(Subject.java:439)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1953)
at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2.run(SparkExecuteStatementOperation.scala:176)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.lang.UnsupportedOperationException: Replacing a view is not supported by catalog: glue_catalog
at org.apache.iceberg.spark.SparkCatalog.replaceView(SparkCatalog.java:686)
at org.apache.spark.sql.execution.datasources.v2.CreateV2ViewExec.replaceView(CreateV2ViewExec.scala:103)
at org.apache.spark.sql.execution.datasources.v2.CreateV2ViewExec.run(CreateV2ViewExec.scala:64)
at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result$lzycompute(V2CommandExec.scala:43)
at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result(V2CommandExec.scala:43)
at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.executeCollect(V2CommandExec.scala:49)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:126)
at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:108)
at org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:384)
at org.apache.spark.sql.execution.SQLExecution$.executeQuery$1(SQLExecution.scala:157)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$10(SQLExecution.scala:220)
at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:108)
at org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:384)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$9(SQLExecution.scala:220)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:405)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:219)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:901)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:83)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:74)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:123)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:114)
at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:521)
at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(origin.scala:77)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:521)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:34)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:303)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:299)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:34)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:34)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:497)
at org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:114)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:101)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:99)
at org.apache.spark.sql.Dataset.(Dataset.scala:223)
at org.apache.spark.sql.Dataset$.$anonfun$ofRows$2(Dataset.scala:103)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:901)
at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:99)
at org.apache.spark.sql.SparkSession.$anonfun$sql$4(SparkSession.scala:692)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:901)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:683)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:714)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:745)
at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:651)
at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.org$apache$spark$sql$hive$thriftserver$SparkExecuteStatementOperation$$execute(SparkExecuteStatementOperation.scala:228)
... 16 more
```

### Environment

```markdown
• **OS**: Linux
• **Python**: 3.9.16
• **dbt version**: 1.9.8
• **dbt-spark version**: 1.9.2
• **Spark**: With Hive Thrift Server
• **Catalog**: AWS Glue with Iceberg v2 tables
• **Target**: spark_dev
```

### Additional Context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.