apache / apache/hudi

[SUPPORT] data loss in new base file after compaction

Open
#8,132 11 comments 0 reactions 0 assignees View on GitHub
area:table-service issue:data-loss pre-0.10.0 priority:high
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**Describe the problem you faced**
I encountered data loss in hudi mor table. After compaction, a base file became smaller and lost data. The issue occurred on 2023-01-10 and I am only able to access archived commits.(The parquet and log files for those instants have already been deleted.) I have uploaded all the related commits
[archived_commit.csv](https://github.com/apache/hudi/files/10921045/archived_commit.csv). Here is the timeline of the basefile: **8fb29db6-81da-4455-a08d-ba3e7ee36856-0**, it shows that its size was 116M until L23. However, it became 66M after compaction and lost 890k records.
lineNum | commitTime | actionType | actionState | Plan
---------|---------------|---------------|---------------|---------------------------------------
L16 | 20230110133240 | compaction | REQUESTED | {"baseInstantTime": "20230110130744", "deltaFilePaths": [".8fb29db6-81da-4455-a08d-ba3e7ee36856-0_20230110130744.log.1_53-7076606-255800368"], "dataFilePath": "8fb29db6-81da-4455-a08d-ba3e7ee36856-0_24-7076286-255788001_20230110130744.parquet", "fileId": "8fb29db6-81da-4455-a08d-ba3e7ee36856-0", "partitionPath": "daas_date=2022", "metrics": {"TOTAL_LOG_FILES": 1.0, "TOTAL_IO_READ_MB": 116.0, "TOTAL_LOG_FILES_SIZE": 24243.0, "TOTAL_IO_WRITE_MB": 116.0, "TOTAL_IO_MB": 232.0}, "bootstrapFilePath": null}
L17 | 20230110133240 | compaction | INFLIGHT | {"baseInstantTime": "20230110130744", "deltaFilePaths": [".8fb29db6-81da-4455-a08d-ba3e7ee36856-0_20230110130744.log.1_53-7076606-255800368"], "dataFilePath": "8fb29db6-81da-4455-a08d-ba3e7ee36856-0_24-7076286-255788001_20230110130744.parquet", "fileId": "8fb29db6-81da-4455-a08d-ba3e7ee36856-0", "partitionPath": "daas_date=2022", "metrics": {"TOTAL_LOG_FILES": 1.0, "TOTAL_IO_READ_MB": 116.0, "TOTAL_LOG_FILES_SIZE": 24243.0, "TOTAL_IO_WRITE_MB": 116.0, "TOTAL_IO_MB": 232.0}
L18 | 20230110133240 | commit | COMPLETED | {\"fileId\": \"8fb29db6-81da-4455-a08d-ba3e7ee36856-0\", \"path\": \"daas_date=2022/8fb29db6-81da-4455-a08d-ba3e7ee36856-0_27-7076664-255801644_20230110133240.parquet\", \"prevCommit\": \"20230110130744\", \"numWrites\": 2060123, \"numDeletes\": 0, \"numUpdateWrites\": 99, \"totalWriteBytes\": 122155577, \"totalWriteErrors\": 0, \"partitionPath\": \"daas_date=2022\", \"totalLogRecords\": 99, \"totalLogFiles\": null, \"totalUpdatedRecordsCompacted\": 99, \"numInserts\": 0, \"totalLogBlocks\": 1, \"totalCorruptLogBlock\": 0, \"totalRollbackBlocks\": 0, \"fileSizeInBytes\": 122155577}
L19 | 20230110135603 | commit | REQUESTED |
L20 | 20230110135603 | commit | INFLIGHT | {\"fileId\": \"3c82ba35-b701-4f34-882a-167146036ab3-0\", \"path\": null, \"prevCommit\": \"20230110133240\", \"numWrites\": 0, \"numDeletes\": 0, \"numUpdateWrites\": 108, \"totalWriteBytes\": 0, \"totalWriteErrors\": 0, \"partitionPath\": null, \"totalLogRecords\": 0, \"totalLogFiles\": null, \"totalUpdatedRecordsCompacted\": 0, \"numInserts\": 0, \"totalLogBlocks\": 0, \"totalCorruptLogBlock\": 0, \"totalRollbackBlocks\": 0, \"fileSizeInBytes\": 0}
L21 | 20230110135603 | commit | COMPLETED | {\"fileId\": \"8fb29db6-81da-4455-a08d-ba3e7ee36856-0\", \"path\": \"daas_date=2022/.8fb29db6-81da-4455-a08d-ba3e7ee36856-0_20230110133240.log.1_53-7077163-255815146\", \"prevCommit\": \"20230110133240\", \"numWrites\": 106, \"numDeletes\": 0, \"numUpdateWrites\": 106, \"totalWriteBytes\": 25777, \"totalWriteErrors\": 0, \"partitionPath\": \"daas_date=2022\", \"totalLogRecords\": 0, \"totalLogFiles\": null, \"totalUpdatedRecordsCompacted\": 0, \"numInserts\": 0, \"totalLogBlocks\": 0, \"totalCorruptLogBlock\": 0, \"totalRollbackBlocks\": 0, \"fileSizeInBytes\": 25777}
L22 | 20230110135817 | compaction | REQUESTED | {\"baseInstantTime\": \"20230110133240\", \"deltaFilePaths\": [\".8fb29db6-81da-4455-a08d-ba3e7ee36856-0_20230110133240.log.1_53-7077163-255815146\"], \"dataFilePath\": \"8fb29db6-81da-4455-a08d-ba3e7ee36856-0_27-7076664-255801644_20230110133240.parquet\", \"fileId\": \"8fb29db6-81da-4455-a08d-ba3e7ee36856-0\", \"partitionPath\": \"daas_date=2022\", \"metrics\": {\"TOTAL_LOG_FILES\": 1.0, \"TOTAL_IO_READ_MB\": 116.0, \"TOTAL_LOG_FILES_SIZE\": 25777.0, \"TOTAL_IO_WRITE_MB\": 116.0, \"TOTAL_IO_MB\": 232.0}, \"bootstrapFilePath\": null}
L23 | 20230110135817 | compaction | INFLIGHT | {\"baseInstantTime\": \"20230110133240\", \"deltaFilePaths\": [\".8fb29db6-81da-4455-a08d-ba3e7ee36856-0_20230110133240.log.1_53-7077163-255815146\"], \"dataFilePath\": \"8fb29db6-81da-4455-a08d-ba3e7ee36856-0_27-7076664-255801644_20230110133240.parquet\", \"fileId\": \"8fb29db6-81da-4455-a08d-ba3e7ee36856-0\", \"partitionPath\": \"daas_date=2022\", \"metrics\": {\"TOTAL_LOG_FILES\": 1.0, \"TOTAL_IO_READ_MB\": 116.0, \"TOTAL_LOG_FILES_SIZE\": 25777.0, \"TOTAL_IO_WRITE_MB\": 116.0, \"TOTAL_IO_MB\": 232.0}, \"bootstrapFilePath\": null}
L24 | 20230110135817 | commit | COMPLETED | {\"fileId\": \"8fb29db6-81da-4455-a08d-ba3e7ee36856-0\", \"path\": \"daas_date=2022/8fb29db6-81da-4455-a08d-ba3e7ee36856-0_20-7077198-255819567_20230110135817.parquet\", \"prevCommit\": \"20230110133240\", \"numWrites\": 1169659, \"numDeletes\": 0, \"numUpdateWrites\": 54, \"totalWriteBytes\": 69228459, \"totalWriteErrors\": 0, \"partitionPath\": \"daas_date=2022\", \"totalLogRecords\": 106, \"totalLogFiles\": null, \"totalUpdatedRecordsCompacted\": 106, \"numInserts\": 52, \"totalLogBlocks\": 1, \"totalCorruptLogBlock\": 0, \"totalRollbackBlocks\": 0, \"fileSizeInBytes\": 69228459}

**Expected behavior**

A clear and concise description of what you expected to happen.

**Environment Description**

* Hudi version : 0.7.0

* Spark version :2.4.4

* Hive version :

* Hadoop version :

* Storage (HDFS/S3/GCS..) : S3

* Running on Docker? (yes/no) : no

**Additional [context**]

1. We have set the parameter 'hoodie.compact.inline.max.delta.commits' to '1', which ensures that compaction runs after each delta commit.
2. We are using default storage configurations such as a limitFileSize of 120M and a parquetBlockSize of 120M.
3. There were no rollbacks or deletes between these commits.
4. This issue has also occurred with other basefiles such as **b244c458-61cb-4535-9f4d-47c64d0cb169-0**, **9241d114-332c-4336-bfd3-7a5345a87159-0**, **724428c7-a600-41a0-887f-feb491fe8c69-0** ...
5. One clue is that seems this issue occurred on basefiles when their size exceeded 100M.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with archived_commit.csv and the L16–L24 timeline, then inspect Hudi 0.7.0 compaction behavior under the listed Spark, S3, and storage-size settings. Compare the affected base files and determine why records disappear when the base file exceeds 100M; the issue is done when the cause and a reproducible validation are established.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.