aws-samples / aws-samples/dbt-glue

Delete not supported

Open
#161 0 comments 2 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
147
Forks
96
Avg merge
7h 4m
Merged PRs (30d)
5

Description

### Describe the bug
After running dbt with the basic command 'dbt run' the model i have prepared with a pre-hook cannot be built
The pre-hook query is a simple delete: delete from schema.table where date_column >= current_date
The error is this:
AnalysisException: DELETE is only supported with v2 tables

I've searched and the v2 tables should refer to a Spark class: https://www.waitingforcode.com/apache-spark-sql/what-new-apache-spark-3-delete-update-merge-api-support/read
![image](https://user-images.githubusercontent.com/127388103/227245101-c49cd3c3-f31e-4aec-aa04-d487b99e7c44.png)

But the version that Glue uses should be Spark 3.1.2 (on dbt i forced the glue_version = "3.0")

### Screenshots and log output
![image](https://user-images.githubusercontent.com/127388103/227229859-10cf56be-1dea-46d5-92c5-9f32e674d174.png)

### System information
**The output of `dbt --version`:**

![image](https://user-images.githubusercontent.com/127388103/227230120-89bdf273-21ff-445b-9b3e-7844dccf4500.png)

**The operating system you're using:**
Windows 10
**The output of `python --version`:**
Python 3.10.10

Thanks in advance for your help, sorry if my explaination is not clear enough.

Contributor guide

Open the contributing guide

Research direction

Reproduce the reported `dbt run` with the pre-hook `delete from schema.table where date_column >= current_date` on Glue 3.0, using the provided system information. Investigate the adapter's pre-hook execution and Spark table compatibility; done means the delete runs without the `DELETE is only supported with v2 tables` error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, sql
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.