apache / apache/druid

doubleMean ignores useDefaultValueForNull=false, treats nulls as zeros

Open
#11,219 1 comment 1 reaction 0 assignees View on GitHub
Area - Null Handling
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

### Affected Version

Seen in 0.20.0

### Description

doubleMean ignores the setting useDefaultValueForNull=false, and treats nulls as zeros. Taking a mean, it possibly ignores the nulls when summing, but apparently divides by a count that includes the rows with nulls. It should exclude those from the count when dividing.

The attached ingestion spec and data file load 10 rows, with myMetric including values from 1 to 10, excluding 5 and 10 (which are null), so the data ends up looking like this:

```
myDim myMetric
1 1
2 2
3 3
4 4
5 null
6 6
7 7
8 8
9 9
10 null
```

Here's a very basic doubleMean query:

```
{
"queryType": "timeseries",
"dataSource": {
"type": "table",
"name": "numbersAndNulls"
},
"intervals": {
"type": "intervals",
"intervals": [
"-146136543-09-08T08:23:32.096Z/146140482-04-24T15:36:27.903Z"
]
},
"granularity": {
"type": "all"
},
"aggregations": [
{
"type": "doubleMean",
"name": "doubleMean",
"fieldName": "myMetric",
"expression": null
}
]
}
```

The mean should be (sum myMetric)/(count myMetric) - 40/8, or 5. However, 4 is returned - 40/10. So it's treating the "5" and "10" rows as if they included 0's.

If we filter for not null, we get the right result.

```
{
"queryType": "timeseries",
"dataSource": {
"type": "table",
"name": "numbersAndNulls"
},
"intervals": {
"type": "intervals",
"intervals": [
"-146136543-09-08T08:23:32.096Z/146140482-04-24T15:36:27.903Z"
]
},
"granularity": {
"type": "all"
},
"filter": {
"type": "not",
"field": {
"type": "selector",
"dimension": "myMetric",
"value": null,
"extractionFn": null
}
},
"aggregations": [
{
"type": "doubleMean",
"name": "doubleMean",
"fieldName": "myMetric",
"expression": null
}
]
}
```

But we shouldn't have to add this filter.

[data.txt](https://github.com/apache/druid/files/6444381/data.txt)
[ingest.txt](https://github.com/apache/druid/files/6444382/ingest.txt)

Contributor guide

Open the contributing guide

Research direction

Start by locating the Java implementation and tests for the doubleMean aggregation, then reproduce the issue with the attached data and ingestion specifications. Verify that null metric rows are excluded from the divisor when useDefaultValueForNull=false, so the sample returns 5 rather than 4, without requiring a not-null filter.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data, databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.