apache / apache/hudi

[SUPPORT] - Data loss after 3 days following upgrade from Hudi 0.11.1 to 0.14.0

Open
#11,959 6 comments 0 reactions 0 assignees View on GitHub
issue:data-loss priority:critical
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**_Tips before filing an issue_**

**Describe the problem you faced**

A clear and concise description of the problem.

We have a COW table which is updated via an UPSERT operation through a Glue Job; the operations were initially performed on Hudi 0.11.1. Moreover the table is partitioned by year, month and day.

Some days after upgrading to Hudi 0.14.0, we noticed that we were having less rows for partitions starting from the update date. Moreover, we noticed that records for a given partition day were dropped with a delay of 3 days. This behaviour was observed when counting the records by partition using Glue or Athena.

On another hand, we also have a Redshift Spectrum subscription built from this table, and when doing the row count check, we could see the "correct" number of rows. However, we could see duplicated data.

Furthermore, we upgraded 4 tables from Hudi 0.11.1 to Hudi 0.14.0 and only with this table we observed such behaviour.

**To Reproduce**

Steps to reproduce the behavior:

1. Table in Hudi 0.11.1
2. Upgrade to Hudi 0.14.0
3. Wait 3 days to observe the data loss.

These are the write configurations set by us.

![Screenshot 2024-06-21 at 13 12 16](https://github.com/user-attachments/assets/b3136357-159e-40fb-be3a-07f7c98cfc80)

**Expected behavior**

Could you please shed some light on why this could have happened?

We should see the correct number of rows in Athena / Glue.

**Environment Description**

* Hudi version : 0.14.0

* Spark version : 3.3.0 (Glue 4)

* Hive version :

* Hadoop version :

* Storage (HDFS/S3/GCS..) : S3

* Running on Docker? (yes/no) : No

**Additional context**

Add any other context about the problem here.

**Stacktrace**

```Add the stacktrace of the error.```

Contributor guide

No contributing guide indexed for this repository

Research direction

The report names a Hudi 0.11.1-to-0.14.0 upgrade, Glue/Spark 3.3.0, S3, and Athena/Spectrum, but no repository files, tests, or entry points. Start by reviewing the supplied write configuration and comparing partition contents and row counts across those readers. Done would require a confirmed cause and a targeted fix or documented resolution.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws
Domain
cloud, data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.