apache / apache/druid

Imperfect rollup in native ingestion when using integer dimensions and replacing nulls with default value

Open
#10,004 1 comment 0 reactions 0 assignees View on GitHub
Area - Batch Ingestion Bug
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

### Affected Version
0.18.0

### Description
During native ingestion `RollupFactsHolder` structure is used for maintaining a sorted list of intermediate rows. It treats missing integer values as nulls and puts them before any other value. Although later when an incremental persist is created, by default null values get replaced with 0. This breaks the sorted ordering of rows which prevents merging of identical rows as the next stage algorithm in `RowCombiningTimeAndDimsIterator` relies on incremental persists being sorted when performing the merge.

### Steps To Reproduce
Import two files below with the given ingestion spec (replacing ``).
The expected result is 3 rows total, although 4 rows are produced.

Input file 1:
```
{"time": 1589512112, "dim": -101, "value": 1}
{"time": 1589512112, "dim": -100, "value": 2}
```
Input file 2:
```
{"time": 1589512112, "dim": null, "value": 10}
{"time": 1589512112, "dim": -100, "value": 20}
```
Ingestion spec:
```
{
"type": "index_parallel",
"spec": {
"ioConfig": {
"type": "index_parallel",
"inputSource": {
"type": "local",
"filter": "*",
"baseDir": ""
},
"inputFormat": {
"type": "json"
}
},
"tuningConfig": {
"type": "index_parallel",
"partitionsSpec": {
"type": "hashed",
"numShards": 1,
"partitionDimensions": [
"dim"
]
},
"forceGuaranteedRollup": true,
"maxNumConcurrentSubTasks": 2,
"splitHintSpec" : {
"type" : "maxSize",
"maxSplitSize" : 1
}
},
"dataSchema": {
"dataSource": "int_rollup_issue",
"granularitySpec": {
"type": "uniform",
"queryGranularity": "HOUR",
"rollup": true,
"intervals": [
"2020-05-15/2020-05-16"
],
"segmentGranularity": "DAY"
},
"timestampSpec": {
"column": "time",
"format": "posix"
},
"dimensionsSpec": {
"dimensions": [
{
"type": "long",
"name": "dim"
}
]
},
"metricsSpec": [
{
"name": "count",
"type": "count"
},
{
"name": "sum_value",
"type": "longSum",
"fieldName": "value"
}
]
}
}
}
```

Contributor guide

Open the contributing guide

Research direction

Reproduce the native-ingestion case with the two JSON files and the provided ingestion spec, confirming that four rows appear instead of three. Read RollupFactsHolder and RowCombiningTimeAndDimsIterator to trace how integer nulls and default values affect ordering and merging. Done means the reproduced input produces three rows with identical rows combined correctly.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.