apache / apache/incubator-xtable
_delta_log folder to be considered as partition when checkpoint parquet files present by hudi reader without metadata table
- Dominant language
- Java
- Stars
- 1.2k
- Forks
- 212
- Avg merge
- 4d 9h
- Merged PRs (30d)
- 16
Description
### Search before asking
- [x] I had searched in the [issues](https://github.com/apache/incubator-xtable/issues?q=is%3Aissue) and found no similar issues.
### Please describe the bug 🐞
turns out when `_delta_log` has checkpoint parquet files such `00000000000000000150.checkpoint.parquet`, then hudi fs backed metadata consider them as hudi partition and fails hard.
In hudi the metadata table is not the source of truce and it sometimes gets corrupted. As a result one can rebuild but having delta logs folder leads corrupt the hudi, unable to read/write.
I have mitigated the problem in our hudi fork by skipping both metadata/_delta_log folders, but i can't see any pure xtable mitigation so far, except take care of purging both _delta_log and metadata folder when recreating the MDT
### Are you willing to submit PR?
- [x] I am willing to submit a PR!
- [ ] I am willing to submit a PR but need help getting started!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the Hudi fs-backed metadata failure with `_delta_log/00000000000000000150.checkpoint.parquet` and without a metadata table. Inspect how the `_delta_log` and `metadata` folders are classified during table reads and writes; done means checkpoint files are not treated as Hudi partitions and the table remains readable and writable.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100