apache / apache/hudi

Help understanding how compaction and normal write work

Open
#17,960 1 comment 0 reactions 0 assignees View on GitHub
type:community-support
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

Hi,

I would like to understand how normal write works after compaction is scheduled.

As we know,during the Hudi compaction schedule phase, Hudi will collect which file slices will be compacted during compaction execution phase.

Before compaction execution phase actullay starts, write operations continue, I would ask where will the write operations write data to after the file slices are marked as to be compacted? to new file groups?

Thanks.

Contributor guide

No contributing guide indexed for this repository

Research direction

The issue names Hudi's compaction schedule and execution phases and asks how normal writes behave after file slices are marked for compaction. No files or tests are identified; start by tracing those phases and the normal write path, and document which file groups receive writes before execution begins.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.