apache / apache/hudi

Support State Compatibility for MergeOnReadInputSplit

Open
#18,414 0 comments 0 reactions 0 assignees View on GitHub
type:devtask
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Task Description

**What needs to be done:**

Allowing modifications to MergeOnReadInputSplit without affecting task recovery from state.

**Why this task is needed:**

`inputSplitsState` in `StreamReadMonitoringFunction` uses the fallback kryo serializer for the state serialization, which doesn't support schema evolution for state. If we modify the attributes in MergeOnReadInputSplit, the flink job can not recover from state.

### Task Type

Code improvement/refactoring

### Related Issues

**Parent feature issue:** #18103
**Related issues:**
NOTE: Use `Relationships` button to add parent/blocking issues after issue is created.

Contributor guide

No contributing guide indexed for this repository

Research direction

Locate MergeOnReadInputSplit and StreamReadMonitoringFunction, then inspect inputSplitsState and its serializer configuration. Verify that changing MergeOnReadInputSplit attributes remains recoverable from existing Flink state, with regression coverage for state restoration.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
stream-processing
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.