Need a way to be able to schedule cleaner service inline and execute as a offline job
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
With the current hudi version 0.13.1 there is no way to schedule cleaner service as part of the writer job. Only possible options are execute inline or scheduleAndExecute offline jobs.
Having an inline schedule only option similar to compaction jobs will be helpful to see when the cleaner services are required.
Related compactor code - [https://github.com/apache/hudi/blob/51ddf1affcdead2e3b5e871ba4816c71e6f4b99a/hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieCompactor.java#L194]
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-6535
- Type: New Feature
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the referenced HoodieCompactor.java around line 194 and compare its scheduling behavior with the current cleaner and writer-job paths in Hudi 0.13.1. Trace how cleaner execution is currently triggered, then define the change as supporting inline scheduling without immediate execution and offline execution; done means both requested modes are available and covered by tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100