MutableSegmentImpl.index() can leave segment in corrupted state when encountering exception
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 2d 55m
- Merged PRs (30d)
- 182
Description
When `MutableSegmentImpl.index()` encounters exception, some columns' dictionary/forward index might already be updated while others are not. This will leave segment in a corrupted state, and eventually fail the segment creation when sealing the segment.
The proper way to handle this is to capture the exception, but continue finishing the entire row. When the input value cannot be determined (e.g. got exception during value transform), we need to fill some dummy value in order to proceed.
Contributor guide
Research direction
Start in MutableSegmentImpl.index() and trace how an exception during value transformation affects dictionary and forward-index updates for the rest of the row. Check the segment-creation and sealing path to understand the failure, then verify that rows with such exceptions finish consistently and segments can be sealed without corruption.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 50/100