Testing "hoodie.read.timeline.holes.resolution.policy" option "USE_TRANSITION_TIME" to resolve "hollow commit" error
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
### Describe the problem you faced
**Issue**: Reading from upstream hudi lake with incremental query type in prod, got "HoodieException: Found hollow commit" error.
Reason: Upstream hudi lake job failed or still running when access to it.
**Environment**: EMR 7.1.0, hudi version 0.14.1, Spark version 3.5
**Work flow**:
1. In the first time run, use `snapshot` to read upstream hudi lake, generate a timestamp as extractId so next we use it to fetch the new data
2. Then twice a week to read the new/updated data with `incremental` type(pass extractId to "BEGIN_INSTANTTIME", generate new extractId)
Hudi suggested to set up `hoodie.read.timeline.holes.resolution.policy` to handle the `hollow commit` error, but for those policy options
1. `Fail`: this the default behaviour, so won't fix the issue
2. `BLOCK`: The upstream lake generating job normally run over hours, so that means the hollow commit will stay there for hours and use this option, we need to wait until the job is done. We cannot wait that longer because we have downstream job need to run as well.
3. `USE_TRANSITION_TIME`: This option will allow us to read the completed commit instead of inflight commit. But as we discussed in the hudi weekly meeting, we were told this option is a new introduced feature, so don't know if it can fix the issue we have.
We have been told bring this question up so hudi team can dig into it to find out the solution. We would like to have a solution that can fix the hollow commit without any data lose, and would like to know if we can switch to the solution without regenerating our whole lake.
Thanks,
Cheers,
### To Reproduce
1. Run a job(producer) to generate a new hudi lake without any failure(No any inflight commit)
2. Run another job(consumer) to read the hudi lake by using `snapshot` query type. Marked the timestamp.
3. Kick off a new producer job to keep it running or make it failed to generate`deltacommit.requested` file under the `.hoodie` folder
4. Run another consumer job to read that hudi lake with `incremental` query type, and pass the marked timestamp in step 2 to `BEGIN_INSTANTTIME`
5.To see if the consumer job failed with "hollow commit" error.
### Expected behavior
After setting up `hoodie.read.timeline.holes.resolution.policy` to `USE_TRANSITION_TIME`
1. Make sure there are no error when reading the hudi lake with `incremental` query type
2. Make sure there are no data lose between different consequential runs
### Environment Description
* Hudi version: 0.14.1
* Spark version: 3.5
* Flink version:
* Hive version:
* Hadoop version:
* Storage (HDFS/S3/GCS..):
* Running on Docker? (yes/no):
### Additional context
_No response_
### Stacktrace
```shell
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the five-step workflow against a Hudi lake, using snapshot and incremental queries with BEGIN_INSTANTTIME and a requested or inflight commit under .hoodie. Test hoodie.read.timeline.holes.resolution.policy=USE_TRANSITION_TIME on Spark 3.5 with Hudi 0.14.1; done means incremental reads avoid the hollow commit error and consecutive runs show no data loss.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- spark
- Domain
- data-engineering, stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100