[to be discussed] Should not archive latest instant with Deltastreamer checkpoint
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
### Task Description
**What needs to be done:**
An ingestion write to a HUDI datasets may add a "checkpoint" information in the commit/replacecommit metadata in HUDI-defined/internal fields like "deltastreamer checkpoint". Regardless of clean/archival config or the amount of instants/time since the latest insert/upsert/bulk_insert/insert_overwrite instant , a user should be able to:
- Rely on HUDI streamer to automatically get the previous "deltastreamer checkpoint" value
Specifically, we want to avoid a scenario where this is a backfill of "non-ingestion" writes (such as clustering), and a subsequent archival causes all ingestion write instants to be deleted from the active timeline.
**Why this task is needed:**
The checkpoint info may be "lost" when the following scenario happens (for table version 6):
1. A backfill of other writes, such as clustering, creates many instants on timeline
2. Archival runs and archives the latest ingestion instant
3. The "checkpoint" info is no longer in the active timeline
We have encountered this scenario in our incremental ingestion workloads, where upon "losing" the checkpoint in the active timeline we need to manually intervene to add it again, in order for incremental reader to infer the checkpoint to resume from.
In addition, archiving all ingestion instants on the timeline also means that the `extraMetadata` field from latest ingestion instant is no longer available.
**Suggested approach**
Currently in our organization's internal (table version 6) HUDI build we have prevented this issue by
- Ensuring archival is blocked on latest "ingestion" write with checkpoint info. We internally define "ingestion" as any deltacommit/commit/replacecommit that is not a clustering, compaction, or delete_partition. But an alternative approach (that we can consider if we upstream this change) might instead be to check if the commit metadata has the deltastreamer checkpoint field.
- To avoid archival being blocked by a lack of ingestion activity, we added an "empty commit" HUDI API (which transfers over the deltastreamer checkpoint field and `extraMetadata` filed) https://github.com/apache/hudi/pull/11606 . Specifically, it will check if some "conditions" are met, and if so, create this empty commit . Currently our "conditions" involve checking if there has been a day since last ingestion commit. But when we upstream this change, we can instead check if archival is blocked due to no ingestion commit. We can add this API to `BaseHoodieWriteClient` and also update deltastreamer API to call it as well.
### Task Type
Code improvement/refactoring
### Related Issues
**Parent feature issue:** (if applicable )
**Related issues:**
NOTE: Use `Relationships` button to add parent/blocking issues after issue is created.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.