apache / apache/hudi

hoodie.properties.backup file does not exist

Open
#11,510 8 comments 0 reactions 0 assignees View on GitHub
area:upgrade
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

Env:
Hudi:0.14
Flink:1.6
CHD:6.32
HDFS:3.0.0
Hive:2.1.1
Action: Search hudi table
We are currently using version Hudi 0.14, which was upgraded from 0.13. We did not encounter this issue in version 0.13.
Specific Issue: I have a streaming task table that runs in production without making any changes to the table or the task, and the following exception is reported:

2024-06-25 01:28:10,582 WARN org.apache.hudi.common.table.HoodieTableConfig [] - Invalid properties file hdfs://10.0.5.131:8020/user/ods/ods_pom_in_transit_time_config/.hoodie/hoodie.properties: {}

2024-06-25 01:28:10,586 WARN org.apache.hudi.common.table.HoodieTableConfig [] - Could not read properties from hdfs://10.0.5.131:8020/user/ods/ods_pom_in_transit_time_config/.hoodie/hoodie.properties.backup: java.io.FileNotFoundException: File does not exist: /user/ods/ods_pom_in_transit_time_config/.hoodie/hoodie.properties.backup

The current streaming task deployment model is Session, and this issue occurs occasionally.

I have the following initial doubts:
1.Without involving any changes to table structure, index, partition, etc., why is the hoodie.properties file cleared?
2.Why was the corresponding hoodie.properties.backup file not generated before the hoodie.properties operation?

It's not just a single table experiencing this issue; currently, we have nearly 30 tables in production with similar problems. This issue frequently occurs in our development environment as well. We have not yet identified the specific cause of this problem.

Thanks!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the HoodieTableConfig warnings and the .hoodie/hoodie.properties and hoodie.properties.backup paths shown in the report. Trace how the Session deployment reads or updates these files, comparing Hudi 0.13 and 0.14 behavior where relevant. Done means identifying a reproducible cause for the missing or cleared properties file, or documenting the conditions needed to reproduce it.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, databases, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.