apache / apache/hudi

Re-populate completed instant from inflight/requested for an empty instant during archival

Open
#15,119 0 comments 0 reactions 0 assignees View on GitHub
component:archiving from-jira priority:high type:devtask
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 4h
Merged PRs (30d)
112

Description

[https://github.com/apache/hudi/pull/5261]

 

Due to process crash, there are chances that a completed instant is empty in the timeline. During archival this was causing issues and so we put in a fix to create a empty instant and add it to archival for book keeping purpose. But strictly speaking, we could re-populate the completed instant from inflight/requested (best effort) so that we can get back the plan as is. Creating a follow up ticket for this fix.

 

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-3851
- Type: Task

Contributor guide

No contributing guide indexed for this repository

Research direction

Review PR #5261 and the Hudi-3851 task for the existing archival fix and the intended follow-up behavior. Trace how an empty completed instant relates to the inflight/requested instants, then verify that archival preserves the plan when it can be recovered and retains the empty instant when it cannot.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.