apache / apache/hudi

Support async compaction in spark DF write()

Open
#14,681 0 comments 0 reactions 0 assignees View on GitHub
area:table-service component:compaction engine:spark from-jira priority:high type:feature
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

spark.write().format("hudi").option(operation, "run_compact") to run compaction

 

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-1329
- Type: New Feature
- Affects version(s):
- 0.9.0

---

## Comments

09/Oct/20 10:04;309637554;[~vbalaji]  hello, i am interested in this issue .  but  I have some doubts , what is the different between  "hoodie.datasource.compaction.async.enable" param and this issue? :D;;;

---

04/Dec/20 03:34;vbalaji;[~309637554]: This API allows only running compaction. Note that there is no input dataframe to be ingested. You can create a dummy dataframe if needed but the operation does not have to care about input DF. It only needs to run compaction (specific compaction id if provided by user) or the oldest one if not provided,;;;

---

22/Jan/21 14:01;shivnarayan;[~vbalaji]: wondering you this is called as async compaction instead of inline compaction ? the ask is to trigger an explicit compaction right? ;;;

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at the Spark DataFrame write path for format("hudi") and trace how the operation option is handled. Compare the existing hoodie.datasource.compaction.async.enable behavior with the requested run_compact operation. Done means compaction can be triggered without an input dataframe, using a supplied compaction ID or the oldest available compaction when none is supplied.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend-api-design, data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.