Support State Compatibility for MergeOnReadInputSplit
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
### Task Description
**What needs to be done:**
Allowing modifications to MergeOnReadInputSplit without affecting task recovery from state.
**Why this task is needed:**
`inputSplitsState` in `StreamReadMonitoringFunction` uses the fallback kryo serializer for the state serialization, which doesn't support schema evolution for state. If we modify the attributes in MergeOnReadInputSplit, the flink job can not recover from state.
### Task Type
Code improvement/refactoring
### Related Issues
**Parent feature issue:** #18103
**Related issues:**
NOTE: Use `Relationships` button to add parent/blocking issues after issue is created.
Contributor guide
No contributing guide indexed for this repository
Research direction
Locate MergeOnReadInputSplit and StreamReadMonitoringFunction, then inspect inputSplitsState and its serializer configuration. Verify that changing MergeOnReadInputSplit attributes remains recoverable from existing Flink state, with regression coverage for state restoration.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- stream-processing
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100