[SUPPORT] Wrong file path for commit-marker-file in Consistent-Bucket-Index layout
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
In Consistent-Bucket-Index file layout:
there are two type of file for each versioned bucket metadata:
1. `{instant}.hashing_metadata`: created in each bucket resizing.
2. `{instant}.commit`: created after `instant` is completed.
All these two type of files should all be stored in this path:
When `loadMetadata`, will load all these two type of files from above path.
But, I find that the `{instant}.commit` file will not be stored in this hash-metadata path.
This file will be stored in data path, which is `{basePath}/{partitionPath}/{instant}.commit`. So when `ConsistentBucketIndexUtils` load metadata for a partition, all files of this type will not be loaded because they are stored in the wrong location.
For example:
> .hashing_metadata

> .commit

**_Tips before filing an issue_**
- Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)?
- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.
- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.
**Describe the problem you faced**
A clear and concise description of the problem.
**To Reproduce**
Steps to reproduce the behavior:
1.
4.
5.
6.
**Expected behavior**
A clear and concise description of what you expected to happen.
**Environment Description**
* Hudi version :
* Spark version :
* Hive version :
* Hadoop version :
* Storage (HDFS/S3/GCS..) :
* Running on Docker? (yes/no) :
**Additional context**
Add any other context about the problem here.
**Stacktrace**
```Add the stacktrace of the error.```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with ConsistentBucketIndexUtils and its loadMetadata path, then trace where the .commit file is written for the Consistent-Bucket-Index layout. Compare that location with the .hashing_metadata location described in the issue. Done means commit-marker files are stored where metadata loading expects them and are included when loading a partition.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100