aws-samples / aws-samples/dbt-glue

--full-refresh (or) -f is not working for Apache HUDI incremental models

Open
#159 2 comments 0 reactions 1 assignee Claimed by @menuetb View on GitHub
bug
Dominant language
Python
Stars
147
Forks
96
Avg merge
7h 4m
Merged PRs (30d)
5

Description

### Describe the bug
A clear and concise description of what the bug is. What command did you run? What happened?
Even if the `-f` flag is added - the models always run in incremental mode and do not recreate the tables! Is there any additional configuration required to enable the full-refresh capabilities for Apache HUDI tables?

### Steps To Reproduce
In as much detail as possible, please provide steps to reproduce the issue. Sample data that triggers the issue, example model code, etc is all very helpful here.

### Expected behavior
A clear and concise description of what you expected to happen.

### Screenshots and log output
If applicable, add screenshots or log output to help explain your problem.

### System information
**The output of `dbt --version`:**
```
Core:
- installed: 1.3.2
- latest: 1.4.5 - Update available!

Your version of dbt-core is out of date!
You can find instructions for upgrading here:
https://docs.getdbt.com/docs/installation

Plugins:
- glue: 1.3.12 - Up to date!
- spark: 1.3.1 - Update available!

At least one plugin is out of date or incompatible with dbt-core.
You can find instructions for upgrading here:
https://docs.getdbt.com/docs/installation
```

**The operating system you're using:**
Linux (Ubuntu)

**The output of `python --version`:**
```
Python 3.8.16
```

### Additional context
Add any other context about the problem here.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.