apache / apache/hudi

[SUPPORT] Wrong file path for commit-marker-file in Consistent-Bucket-Index layout

Open
#12,338 0 comments 0 reactions 0 assignees View on GitHub
area:index
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

In Consistent-Bucket-Index file layout:
there are two type of file for each versioned bucket metadata:
1. `{instant}.hashing_metadata`: created in each bucket resizing.
2. `{instant}.commit`: created after `instant` is completed.
image

All these two type of files should all be stored in this path:
image
When `loadMetadata`, will load all these two type of files from above path.
image
But, I find that the `{instant}.commit` file will not be stored in this hash-metadata path.
image
This file will be stored in data path, which is `{basePath}/{partitionPath}/{instant}.commit`. So when `ConsistentBucketIndexUtils` load metadata for a partition, all files of this type will not be loaded because they are stored in the wrong location.

For example:
> .hashing_metadata

![image](https://github.com/user-attachments/assets/83288d9f-f22d-40d1-a3a7-15704a01524e)

> .commit

![image](https://github.com/user-attachments/assets/3b8308fe-7241-4746-8bf6-c33a4e4cdd24)

**_Tips before filing an issue_**

- Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)?

- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.

- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.

**Describe the problem you faced**

A clear and concise description of the problem.

**To Reproduce**

Steps to reproduce the behavior:

1.
4.
5.
6.

**Expected behavior**

A clear and concise description of what you expected to happen.

**Environment Description**

* Hudi version :

* Spark version :

* Hive version :

* Hadoop version :

* Storage (HDFS/S3/GCS..) :

* Running on Docker? (yes/no) :

**Additional context**

Add any other context about the problem here.

**Stacktrace**

```Add the stacktrace of the error.```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with ConsistentBucketIndexUtils and its loadMetadata path, then trace where the .commit file is written for the Consistent-Bucket-Index layout. Compare that location with the .hashing_metadata location described in the issue. Done means commit-marker files are stored where metadata loading expects them and are included when loading a partition.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.