apache / apache/pinot

MutableSegmentImpl.index() can leave segment in corrupted state when encountering exception

Open
#16,316 9 comments 0 reactions 0 assignees View on GitHub
bug ingestion real-time
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 55m
Merged PRs (30d)
182

Description

When `MutableSegmentImpl.index()` encounters exception, some columns' dictionary/forward index might already be updated while others are not. This will leave segment in a corrupted state, and eventually fail the segment creation when sealing the segment.

The proper way to handle this is to capture the exception, but continue finishing the entire row. When the input value cannot be determined (e.g. got exception during value transform), we need to fill some dummy value in order to proceed.

Contributor guide

Open the contributing guide

Research direction

Start in MutableSegmentImpl.index() and trace how an exception during value transformation affects dictionary and forward-index updates for the rest of the row. Check the segment-creation and sealing path to understand the failure, then verify that rows with such exceptions finish consistently and segments can be sealed without corruption.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.