apache / apache/hudi

Provide accurate numDeletes in commit metadata

Open
#15,471 0 comments 0 reactions 1 assignee Assigned to @yihua View on GitHub
from-jira priority:high type:bug
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

When doing a simple computation of {{numInserts - numDeletes}} for all the commits, this leads to negative total records.  Need to check if number of inserts and deletes are accurate when both inserts and deletes exist in the same input batch for upsert.

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-4958
- Type: Bug
- Fix version(s):
- 1.1.0

---

## Comments

30/Sep/22 21:26;guoyihua;I check the commit data after insert, upsert (including deletes with "_hoodie_is_deleted"), and delete operations using Spark datasource in the Spark Guide.  The numInserts and numDeletes look accurate.  Also the logic for deriving numDeletes in HoodieMergeHandle looks OK.  We need to see if the inaccuracy comes from the custom payload implementation.;;;

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.