apache / apache/incubator-xtable

_delta_log folder to be considered as partition when checkpoint parquet files present by hudi reader without metadata table

Open
#813 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
1.2k
Forks
212
Avg merge
4d 9h
Merged PRs (30d)
16

Description

### Search before asking

- [x] I had searched in the [issues](https://github.com/apache/incubator-xtable/issues?q=is%3Aissue) and found no similar issues.

### Please describe the bug 🐞

turns out when `_delta_log` has checkpoint parquet files such `00000000000000000150.checkpoint.parquet`, then hudi fs backed metadata consider them as hudi partition and fails hard.

In hudi the metadata table is not the source of truce and it sometimes gets corrupted. As a result one can rebuild but having delta logs folder leads corrupt the hudi, unable to read/write.

I have mitigated the problem in our hudi fork by skipping both metadata/_delta_log folders, but i can't see any pure xtable mitigation so far, except take care of purging both _delta_log and metadata folder when recreating the MDT

### Are you willing to submit PR?

- [x] I am willing to submit a PR!
- [ ] I am willing to submit a PR but need help getting started!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the Hudi fs-backed metadata failure with `_delta_log/00000000000000000150.checkpoint.parquet` and without a metadata table. Inspect how the `_delta_log` and `metadata` folders are classified during table reads and writes; done means checkpoint files are not treated as Hudi partitions and the table remains readable and writable.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.