Support async compaction in spark DF write()
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
spark.write().format("hudi").option(operation, "run_compact") to run compaction
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-1329
- Type: New Feature
- Affects version(s):
- 0.9.0
---
## Comments
09/Oct/20 10:04;309637554;[~vbalaji] hello, i am interested in this issue . but I have some doubts , what is the different between "hoodie.datasource.compaction.async.enable" param and this issue? :D;;;
---
04/Dec/20 03:34;vbalaji;[~309637554]: This API allows only running compaction. Note that there is no input dataframe to be ingested. You can create a dummy dataframe if needed but the operation does not have to care about input DF. It only needs to run compaction (specific compaction id if provided by user) or the oldest one if not provided,;;;
---
22/Jan/21 14:01;shivnarayan;[~vbalaji]: wondering you this is called as async compaction instead of inline compaction ? the ask is to trigger an explicit compaction right? ;;;
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at the Spark DataFrame write path for format("hudi") and trace how the operation option is handled. Compare the existing hoodie.datasource.compaction.async.enable behavior with the requested run_compact operation. Done means compaction can be triggered without an input dataframe, using a supplied compaction ID or the oldest available compaction when none is supplied.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend-api-design, data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100