hoodie.properties.backup file does not exist
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
Env:
Hudi:0.14
Flink:1.6
CHD:6.32
HDFS:3.0.0
Hive:2.1.1
Action: Search hudi table
We are currently using version Hudi 0.14, which was upgraded from 0.13. We did not encounter this issue in version 0.13.
Specific Issue: I have a streaming task table that runs in production without making any changes to the table or the task, and the following exception is reported:
2024-06-25 01:28:10,582 WARN org.apache.hudi.common.table.HoodieTableConfig [] - Invalid properties file hdfs://10.0.5.131:8020/user/ods/ods_pom_in_transit_time_config/.hoodie/hoodie.properties: {}
2024-06-25 01:28:10,586 WARN org.apache.hudi.common.table.HoodieTableConfig [] - Could not read properties from hdfs://10.0.5.131:8020/user/ods/ods_pom_in_transit_time_config/.hoodie/hoodie.properties.backup: java.io.FileNotFoundException: File does not exist: /user/ods/ods_pom_in_transit_time_config/.hoodie/hoodie.properties.backup
The current streaming task deployment model is Session, and this issue occurs occasionally.
I have the following initial doubts:
1.Without involving any changes to table structure, index, partition, etc., why is the hoodie.properties file cleared?
2.Why was the corresponding hoodie.properties.backup file not generated before the hoodie.properties operation?
It's not just a single table experiencing this issue; currently, we have nearly 30 tables in production with similar problems. This issue frequently occurs in our development environment as well. We have not yet identified the specific cause of this problem.
Thanks!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the HoodieTableConfig warnings and the .hoodie/hoodie.properties and hoodie.properties.backup paths shown in the report. Trace how the Session deployment reads or updates these files, comparing Hudi 0.13 and 0.14 behavior where relevant. Done means identifying a reproducible cause for the missing or cleared properties file, or documenting the conditions needed to reproduce it.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, databases, stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100