apache / apache/hudi

Delete operations not propagating to downstream Hudi table via DeltaStreamer

Open
#13,973 13 comments 0 reactions 1 assignee Claimed by @vaibhavk1992 View on GitHub
type:bug
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Bug Description

**What happened:** I am experiencing an issue where delete operations in a Hudi table do not propagate to another table when using DeltaStreamer.

**What you expected:** I have one main hudi table in which I am setting '_hoodie_is_deleted' while upsert for deletion purpose. This is actually deleting the row from here. I have another pipeline which is a deltaStreamer job and that syncs the changes in this table to another hudi table. The mode of write in both is COW. Inserts and updates are working fine but deletion is not propagating to downstream table.

**Steps to reproduce:**
1. Create hudi table using [options](https://github.com/asija-varun92/cdc-hudi-lakehouse/tree/main/src/hudi_config)
2. Run deltaStreamer job to propagate same changes into another hudi table. [my job](https://github.com/asija-varun92/hudi-data-sync)
3. Try inserting new row into 1st hudi table and run deltaS
treamer job. it will work fine. Then try deleting the same row. It will be deleted from first table bit after running deltaStreamer job the delete operation will not propagate on the downstream table

### Environment

**Hudi version:** 1.0
**Query engine:** Spark 3.5
**Relevant configs:** All configs present in above mentioned github mini projects

### Logs and Stack Trace

_No response_

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.