Suspicious persists during the ingestion
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 58m
- Merged PRs (30d)
- 233
Description
### Affected Version
0.17, 0.18
### Description
```
2020-05-29T19:39:41,325 INFO [[index_kinesis_events_f3f78b11e73489e_cpmeplam]-appenderator-persist] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Flushed in-memory data for segment[events_2020-05-29T17:00:00.000Z_2020-05-29T18:00:00.000Z_2020-05-29T17:00:00.217Z_58] spill[12] to disk in [2,861] ms (99,998 rows).
2020-05-29T19:39:41,354 INFO [[index_kinesis_events_f3f78b11e73489e_cpmeplam]-appenderator-persist] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Flushed in-memory data for segment[events_2020-05-29T16:00:00.000Z_2020-05-29T17:00:00.000Z_2020-05-29T16:00:00.435Z_378] spill[9] to disk in [10] ms (2 rows).
...
```
Observed this pattern of persists was repeatedly happening when `maxRowsInMemory` was 100,000. `maxTotalRows` was default.
Contributor guide
Research direction
Start with AppenderatorImpl and the ingestion settings shown in the report, especially maxRowsInMemory and maxTotalRows; compare the repeated persist log entries with the expected ingestion behavior. Done means determining whether the persists are erroneous and, if so, defining a reproducible failure and regression coverage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100