apache / apache/hudi

Testing "hoodie.read.timeline.holes.resolution.policy" option "USE_TRANSITION_TIME" to resolve "hollow commit" error

Open
#19,693 1 comment 0 reactions 0 assignees View on GitHub
type:community-support
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Describe the problem you faced

**Issue**: Reading from upstream hudi lake with incremental query type in prod, got "HoodieException: Found hollow commit" error.
Reason: Upstream hudi lake job failed or still running when access to it.
**Environment**: EMR 7.1.0, hudi version 0.14.1, Spark version 3.5
**Work flow**:
1. In the first time run, use `snapshot` to read upstream hudi lake, generate a timestamp as extractId so next we use it to fetch the new data
2. Then twice a week to read the new/updated data with `incremental` type(pass extractId to "BEGIN_INSTANTTIME", generate new extractId)

Hudi suggested to set up `hoodie.read.timeline.holes.resolution.policy` to handle the `hollow commit` error, but for those policy options
1. `Fail`: this the default behaviour, so won't fix the issue
2. `BLOCK`: The upstream lake generating job normally run over hours, so that means the hollow commit will stay there for hours and use this option, we need to wait until the job is done. We cannot wait that longer because we have downstream job need to run as well.
3. `USE_TRANSITION_TIME`: This option will allow us to read the completed commit instead of inflight commit. But as we discussed in the hudi weekly meeting, we were told this option is a new introduced feature, so don't know if it can fix the issue we have.

We have been told bring this question up so hudi team can dig into it to find out the solution. We would like to have a solution that can fix the hollow commit without any data lose, and would like to know if we can switch to the solution without regenerating our whole lake.

Thanks,

Cheers,

### To Reproduce

1. Run a job(producer) to generate a new hudi lake without any failure(No any inflight commit)
2. Run another job(consumer) to read the hudi lake by using `snapshot` query type. Marked the timestamp.
3. Kick off a new producer job to keep it running or make it failed to generate`deltacommit.requested` file under the `.hoodie` folder
4. Run another consumer job to read that hudi lake with `incremental` query type, and pass the marked timestamp in step 2 to `BEGIN_INSTANTTIME`
5.To see if the consumer job failed with "hollow commit" error.

### Expected behavior

After setting up `hoodie.read.timeline.holes.resolution.policy` to `USE_TRANSITION_TIME`
1. Make sure there are no error when reading the hudi lake with `incremental` query type
2. Make sure there are no data lose between different consequential runs

### Environment Description

* Hudi version: 0.14.1
* Spark version: 3.5
* Flink version:
* Hive version:
* Hadoop version:
* Storage (HDFS/S3/GCS..):
* Running on Docker? (yes/no):

### Additional context

_No response_

### Stacktrace

```shell

```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the five-step workflow against a Hudi lake, using snapshot and incremental queries with BEGIN_INSTANTTIME and a requested or inflight commit under .hoodie. Test hoodie.read.timeline.holes.resolution.policy=USE_TRANSITION_TIME on Spark 3.5 with Hudi 0.14.1; done means incremental reads avoid the hollow commit error and consecutive runs show no data loss.

Written by the indexing model from the issue text.

Assessment

Tech stack
spark
Domain
data-engineering, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.