[SUPPORT] data loss in new base file after compaction
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
**Describe the problem you faced**
I encountered data loss in hudi mor table. After compaction, a base file became smaller and lost data. The issue occurred on 2023-01-10 and I am only able to access archived commits.(The parquet and log files for those instants have already been deleted.) I have uploaded all the related commits
[archived_commit.csv](https://github.com/apache/hudi/files/10921045/archived_commit.csv). Here is the timeline of the basefile: **8fb29db6-81da-4455-a08d-ba3e7ee36856-0**, it shows that its size was 116M until L23. However, it became 66M after compaction and lost 890k records.
lineNum | commitTime | actionType | actionState | Plan
---------|---------------|---------------|---------------|---------------------------------------
L16 | 20230110133240 | compaction | REQUESTED | {"baseInstantTime": "20230110130744", "deltaFilePaths": [".8fb29db6-81da-4455-a08d-ba3e7ee36856-0_20230110130744.log.1_53-7076606-255800368"], "dataFilePath": "8fb29db6-81da-4455-a08d-ba3e7ee36856-0_24-7076286-255788001_20230110130744.parquet", "fileId": "8fb29db6-81da-4455-a08d-ba3e7ee36856-0", "partitionPath": "daas_date=2022", "metrics": {"TOTAL_LOG_FILES": 1.0, "TOTAL_IO_READ_MB": 116.0, "TOTAL_LOG_FILES_SIZE": 24243.0, "TOTAL_IO_WRITE_MB": 116.0, "TOTAL_IO_MB": 232.0}, "bootstrapFilePath": null}
L17 | 20230110133240 | compaction | INFLIGHT | {"baseInstantTime": "20230110130744", "deltaFilePaths": [".8fb29db6-81da-4455-a08d-ba3e7ee36856-0_20230110130744.log.1_53-7076606-255800368"], "dataFilePath": "8fb29db6-81da-4455-a08d-ba3e7ee36856-0_24-7076286-255788001_20230110130744.parquet", "fileId": "8fb29db6-81da-4455-a08d-ba3e7ee36856-0", "partitionPath": "daas_date=2022", "metrics": {"TOTAL_LOG_FILES": 1.0, "TOTAL_IO_READ_MB": 116.0, "TOTAL_LOG_FILES_SIZE": 24243.0, "TOTAL_IO_WRITE_MB": 116.0, "TOTAL_IO_MB": 232.0}
L18 | 20230110133240 | commit | COMPLETED | {\"fileId\": \"8fb29db6-81da-4455-a08d-ba3e7ee36856-0\", \"path\": \"daas_date=2022/8fb29db6-81da-4455-a08d-ba3e7ee36856-0_27-7076664-255801644_20230110133240.parquet\", \"prevCommit\": \"20230110130744\", \"numWrites\": 2060123, \"numDeletes\": 0, \"numUpdateWrites\": 99, \"totalWriteBytes\": 122155577, \"totalWriteErrors\": 0, \"partitionPath\": \"daas_date=2022\", \"totalLogRecords\": 99, \"totalLogFiles\": null, \"totalUpdatedRecordsCompacted\": 99, \"numInserts\": 0, \"totalLogBlocks\": 1, \"totalCorruptLogBlock\": 0, \"totalRollbackBlocks\": 0, \"fileSizeInBytes\": 122155577}
L19 | 20230110135603 | commit | REQUESTED |
L20 | 20230110135603 | commit | INFLIGHT | {\"fileId\": \"3c82ba35-b701-4f34-882a-167146036ab3-0\", \"path\": null, \"prevCommit\": \"20230110133240\", \"numWrites\": 0, \"numDeletes\": 0, \"numUpdateWrites\": 108, \"totalWriteBytes\": 0, \"totalWriteErrors\": 0, \"partitionPath\": null, \"totalLogRecords\": 0, \"totalLogFiles\": null, \"totalUpdatedRecordsCompacted\": 0, \"numInserts\": 0, \"totalLogBlocks\": 0, \"totalCorruptLogBlock\": 0, \"totalRollbackBlocks\": 0, \"fileSizeInBytes\": 0}
L21 | 20230110135603 | commit | COMPLETED | {\"fileId\": \"8fb29db6-81da-4455-a08d-ba3e7ee36856-0\", \"path\": \"daas_date=2022/.8fb29db6-81da-4455-a08d-ba3e7ee36856-0_20230110133240.log.1_53-7077163-255815146\", \"prevCommit\": \"20230110133240\", \"numWrites\": 106, \"numDeletes\": 0, \"numUpdateWrites\": 106, \"totalWriteBytes\": 25777, \"totalWriteErrors\": 0, \"partitionPath\": \"daas_date=2022\", \"totalLogRecords\": 0, \"totalLogFiles\": null, \"totalUpdatedRecordsCompacted\": 0, \"numInserts\": 0, \"totalLogBlocks\": 0, \"totalCorruptLogBlock\": 0, \"totalRollbackBlocks\": 0, \"fileSizeInBytes\": 25777}
L22 | 20230110135817 | compaction | REQUESTED | {\"baseInstantTime\": \"20230110133240\", \"deltaFilePaths\": [\".8fb29db6-81da-4455-a08d-ba3e7ee36856-0_20230110133240.log.1_53-7077163-255815146\"], \"dataFilePath\": \"8fb29db6-81da-4455-a08d-ba3e7ee36856-0_27-7076664-255801644_20230110133240.parquet\", \"fileId\": \"8fb29db6-81da-4455-a08d-ba3e7ee36856-0\", \"partitionPath\": \"daas_date=2022\", \"metrics\": {\"TOTAL_LOG_FILES\": 1.0, \"TOTAL_IO_READ_MB\": 116.0, \"TOTAL_LOG_FILES_SIZE\": 25777.0, \"TOTAL_IO_WRITE_MB\": 116.0, \"TOTAL_IO_MB\": 232.0}, \"bootstrapFilePath\": null}
L23 | 20230110135817 | compaction | INFLIGHT | {\"baseInstantTime\": \"20230110133240\", \"deltaFilePaths\": [\".8fb29db6-81da-4455-a08d-ba3e7ee36856-0_20230110133240.log.1_53-7077163-255815146\"], \"dataFilePath\": \"8fb29db6-81da-4455-a08d-ba3e7ee36856-0_27-7076664-255801644_20230110133240.parquet\", \"fileId\": \"8fb29db6-81da-4455-a08d-ba3e7ee36856-0\", \"partitionPath\": \"daas_date=2022\", \"metrics\": {\"TOTAL_LOG_FILES\": 1.0, \"TOTAL_IO_READ_MB\": 116.0, \"TOTAL_LOG_FILES_SIZE\": 25777.0, \"TOTAL_IO_WRITE_MB\": 116.0, \"TOTAL_IO_MB\": 232.0}, \"bootstrapFilePath\": null}
L24 | 20230110135817 | commit | COMPLETED | {\"fileId\": \"8fb29db6-81da-4455-a08d-ba3e7ee36856-0\", \"path\": \"daas_date=2022/8fb29db6-81da-4455-a08d-ba3e7ee36856-0_20-7077198-255819567_20230110135817.parquet\", \"prevCommit\": \"20230110133240\", \"numWrites\": 1169659, \"numDeletes\": 0, \"numUpdateWrites\": 54, \"totalWriteBytes\": 69228459, \"totalWriteErrors\": 0, \"partitionPath\": \"daas_date=2022\", \"totalLogRecords\": 106, \"totalLogFiles\": null, \"totalUpdatedRecordsCompacted\": 106, \"numInserts\": 52, \"totalLogBlocks\": 1, \"totalCorruptLogBlock\": 0, \"totalRollbackBlocks\": 0, \"fileSizeInBytes\": 69228459}
**Expected behavior**
A clear and concise description of what you expected to happen.
**Environment Description**
* Hudi version : 0.7.0
* Spark version :2.4.4
* Hive version :
* Hadoop version :
* Storage (HDFS/S3/GCS..) : S3
* Running on Docker? (yes/no) : no
**Additional [context**]
1. We have set the parameter 'hoodie.compact.inline.max.delta.commits' to '1', which ensures that compaction runs after each delta commit.
2. We are using default storage configurations such as a limitFileSize of 120M and a parquetBlockSize of 120M.
3. There were no rollbacks or deletes between these commits.
4. This issue has also occurred with other basefiles such as **b244c458-61cb-4535-9f4d-47c64d0cb169-0**, **9241d114-332c-4336-bfd3-7a5345a87159-0**, **724428c7-a600-41a0-887f-feb491fe8c69-0** ...
5. One clue is that seems this issue occurred on basefiles when their size exceeded 100M.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with archived_commit.csv and the L16–L24 timeline, then inspect Hudi 0.7.0 compaction behavior under the listed Spark, S3, and storage-size settings. Compare the affected base files and determine why records disappear when the base file exceeds 100M; the issue is done when the cause and a reproducible validation are established.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100