apache / apache/druid

Improper Appenderator.add() calls concurrent with persist

Open
#8,492 2 comments 0 reactions 0 assignees View on GitHub
Area - Streaming Ingestion Bug
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

This issue is to track NPE reported in https://github.com/RoaringBitmap/RoaringBitmap/issues/358 but it doesn't seem that it may be caused by a bug in RoaringBitmap. Hence I think it's caused by a bug in Druid.

@jihoonson

> the read operation (persist operation in Druid) happens only when all write operations (indexing operation in Druid) are finished.

Could you demonstrate which synchronization mechanism or a happens-before edge or a casualty edge specifically ensure this? I cannot find it. It seems to happen somewhere in `SeekableStreamIndexTaskRunner.runInternal()` (related: https://github.com/apache/incubator-druid/issues/7360).

Contributor guide

Open the contributing guide

Research direction

Start by reading SeekableStreamIndexTaskRunner.runInternal() and the related issue linked in the report, then trace how indexing writes overlap with Druid's persist operation. Done means either demonstrating the synchronization or happens-before edge that prevents concurrent Appenderator.add() and persist calls, or confirming and documenting the NPE fix path.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.