Delete operations not propagating to downstream Hudi table via DeltaStreamer
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
### Bug Description
**What happened:** I am experiencing an issue where delete operations in a Hudi table do not propagate to another table when using DeltaStreamer.
**What you expected:** I have one main hudi table in which I am setting '_hoodie_is_deleted' while upsert for deletion purpose. This is actually deleting the row from here. I have another pipeline which is a deltaStreamer job and that syncs the changes in this table to another hudi table. The mode of write in both is COW. Inserts and updates are working fine but deletion is not propagating to downstream table.
**Steps to reproduce:**
1. Create hudi table using [options](https://github.com/asija-varun92/cdc-hudi-lakehouse/tree/main/src/hudi_config)
2. Run deltaStreamer job to propagate same changes into another hudi table. [my job](https://github.com/asija-varun92/hudi-data-sync)
3. Try inserting new row into 1st hudi table and run deltaS
treamer job. it will work fine. Then try deleting the same row. It will be deleted from first table bit after running deltaStreamer job the delete operation will not propagate on the downstream table
### Environment
**Hudi version:** 1.0
**Query engine:** Spark 3.5
**Relevant configs:** All configs present in above mentioned github mini projects
### Logs and Stack Trace
_No response_
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.