dbt-labs / dbt-labs/dbt-adapters
[Feature] Replacing a view is not supported by catalog: glue_catalog
- Dominant language
- Python
- Stars
- 233
- Forks
- 362
- Avg merge
- 3d 22h
- Merged PRs (30d)
- 9
Description
### Is this a new bug?
- [x] I believe this is a new bug
- [x] I have searched the existing issues, and I could not find an existing issue for this bug
### Which packages are affected?
- [ ] dbt-adapters
- [ ] dbt-tests-adapter
- [ ] dbt-athena
- [ ] dbt-athena-community
- [ ] dbt-bigquery
- [ ] dbt-postgres
- [ ] dbt-redshift
- [ ] dbt-snowflake
- [x] dbt-spark
### Current Behavior
- Running dbt run --select stg_orders fails with error: java.lang.UnsupportedOperationException: Replacing a view is not supported by catalog: glue_catalog
- The error occurs when dbt attempts to execute CREATE OR REPLACE VIEW against the Glue catalog
- Model is configured with materialized = 'view' in the dbt configuration
### Expected Behavior
- dbt should successfully create/update view models when using Glue catalog
- Either support CREATE OR REPLACE VIEW or implement alternative logic to drop and recreate views
### Steps To Reproduce
## Prerequisites
• AWS environment with Glue catalog configured
• Spark cluster with Iceberg support
• dbt-spark adapter version 1.9.2
• dbt version 1.9.8
• Access to AWS Glue Data Catalog
### Environment Setup
1. Configure dbt profile with Spark connection using Glue catalog:
yaml
# profiles.yml
target: spark_dev
outputs:
spark_dev:
type: spark
method: thrift
host:
port:
# Glue catalog configuration
2. Ensure Glue catalog is using Iceberg format (v2 tables)
### Reproduction Steps
1. Create a simple dbt model with view materialization:
sql
-- models/staging/stg_orders.sql
{{
config(
materialized = 'view'
)
}}
SELECT
order_id,
customer_id,
order_date,
order_amount
FROM {{ source('source_database', 'orders') }}
2. Run the model:
bash
dbt run --select stg_orders
### Relevant log output
```shell
16:26:00 On master: ROLLBACK
16:26:00 Opening a new connection, currently in state init
16:26:00 Spark adapter: NotImplemented: rollback
16:26:00 Spark adapter: NotImplemented: add_begin_query
16:26:00 Spark adapter: NotImplemented: commit
16:26:00 On master: ROLLBACK
16:26:00 Spark adapter: NotImplemented: rollback
16:26:00 On master: Close
16:26:00 Connection 'master' was properly closed.
16:26:00 Connection 'model.aws_dbt_control_framework.stg_orders' was properly closed.
16:26:00
16:26:00 Finished running 1 view model in 0 hours 0 minutes and 7.40 seconds (7.40s).
16:26:00 Command end result
16:26:00 Wrote artifact WritableManifest to /home/ssm-user/aws-dbt-control-framework/target/manifest.json
16:26:00 Wrote artifact SemanticManifest to /home/ssm-user/aws-dbt-control-framework/target/semantic_manifest.json
16:26:00 Wrote artifact RunExecutionResult to /home/ssm-user/aws-dbt-control-framework/target/run_results.json
16:26:00
16:26:00 Completed with 1 error, 0 partial successes, and 0 warnings:
16:26:00
16:26:00 Runtime Error in model stg_orders (models/staging/stg_orders.sql)
Database Error
org.apache.hive.service.cli.HiveSQLException: Error running query: java.lang.UnsupportedOperationException: Replacing a view is not supported by catalog: glue_catalog
at org.apache.spark.sql.hive.thriftserver.HiveThriftServerErrors$.runningQueryError(HiveThriftServerErrors.scala:46)
at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.org$apache$spark$sql$hive$thriftserver$SparkExecuteStatementOperation$$execute(SparkExecuteStatementOperation.scala:263)
at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3.$anonfun$run$2(SparkExecuteStatementOperation.scala:167)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at org.apache.spark.sql.hive.thriftserver.SparkOperation.withLocalProperties(SparkOperation.scala:79)
at org.apache.spark.sql.hive.thriftserver.SparkOperation.withLocalProperties$(SparkOperation.scala:63)
at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.withLocalProperties(SparkExecuteStatementOperation.scala:41)
at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3.run(SparkExecuteStatementOperation.scala:167)
at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3.run(SparkExecuteStatementOperation.scala:162)
at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
at java.base/javax.security.auth.Subject.doAs(Subject.java:439)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1953)
at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2.run(SparkExecuteStatementOperation.scala:176)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.lang.UnsupportedOperationException: Replacing a view is not supported by catalog: glue_catalog
at org.apache.iceberg.spark.SparkCatalog.replaceView(SparkCatalog.java:686)
at org.apache.spark.sql.execution.datasources.v2.CreateV2ViewExec.replaceView(CreateV2ViewExec.scala:103)
at org.apache.spark.sql.execution.datasources.v2.CreateV2ViewExec.run(CreateV2ViewExec.scala:64)
at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result$lzycompute(V2CommandExec.scala:43)
at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result(V2CommandExec.scala:43)
at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.executeCollect(V2CommandExec.scala:49)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:126)
at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:108)
at org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:384)
at org.apache.spark.sql.execution.SQLExecution$.executeQuery$1(SQLExecution.scala:157)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$10(SQLExecution.scala:220)
at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:108)
at org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:384)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$9(SQLExecution.scala:220)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:405)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:219)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:901)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:83)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:74)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:123)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:114)
at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:521)
at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(origin.scala:77)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:521)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:34)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:303)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:299)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:34)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:34)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:497)
at org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:114)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:101)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:99)
at org.apache.spark.sql.Dataset.(Dataset.scala:223)
at org.apache.spark.sql.Dataset$.$anonfun$ofRows$2(Dataset.scala:103)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:901)
at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:99)
at org.apache.spark.sql.SparkSession.$anonfun$sql$4(SparkSession.scala:692)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:901)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:683)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:714)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:745)
at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:651)
at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.org$apache$spark$sql$hive$thriftserver$SparkExecuteStatementOperation$$execute(SparkExecuteStatementOperation.scala:228)
... 16 more
```
### Environment
```markdown
• **OS**: Linux
• **Python**: 3.9.16
• **dbt version**: 1.9.8
• **dbt-spark version**: 1.9.2
• **Spark**: With Hive Thrift Server
• **Catalog**: AWS Glue with Iceberg v2 tables
• **Target**: spark_dev
```
### Additional Context
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.