aws-samples / aws-samples/dbt-glue
Delete not supported
- Dominant language
- Python
- Stars
- 147
- Forks
- 96
- Avg merge
- 7h 4m
- Merged PRs (30d)
- 5
Description
### Describe the bug
After running dbt with the basic command 'dbt run' the model i have prepared with a pre-hook cannot be built
The pre-hook query is a simple delete: delete from schema.table where date_column >= current_date
The error is this:
AnalysisException: DELETE is only supported with v2 tables
I've searched and the v2 tables should refer to a Spark class: https://www.waitingforcode.com/apache-spark-sql/what-new-apache-spark-3-delete-update-merge-api-support/read

But the version that Glue uses should be Spark 3.1.2 (on dbt i forced the glue_version = "3.0")
### Screenshots and log output

### System information
**The output of `dbt --version`:**

**The operating system you're using:**
Windows 10
**The output of `python --version`:**
Python 3.10.10
Thanks in advance for your help, sorry if my explaination is not clear enough.
Contributor guide
Research direction
Reproduce the reported `dbt run` with the pre-hook `delete from schema.table where date_column >= current_date` on Glue 3.0, using the provided system information. Investigate the adapter's pre-hook execution and Spark table compatibility; done means the delete runs without the `DELETE is only supported with v2 tables` error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, sql
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100