apache / apache/druid

Dimension from inputSource is ignored when not explicitly defined

Open
#9,914 3 comments 1 reaction 0 assignees View on GitHub
Area - Batch Ingestion Bug
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

### Affected Version

v0.18.1

### Description

Given ingestion spec:
```
{
"type": "index_parallel",
"spec": {
"dataSchema": {
"dataSource": "visitor_events-sessions-test1",
"timestampSpec": {
"column": "timestamp",
"format": "auto"
},
"dimensionsSpec": {
"dimensions": ["site_id", "environment_name"]
},
"metricsSpec": [
{ "type": "doubleSum", "name": "count", "fieldName": "count" },
{ "type": "thetaSketch", "name": "visitor_id_sketch", "fieldName": "visitor_id" }
],
"granularitySpec": {
"segmentGranularity": "day",
"queryGranularity": "hour",
"intervals": [
"2020-05-19/2020-05-21"
]
}
},
"ioConfig": {
"type": "index_parallel",
"inputSource": {
"type": "druid",
"dataSource": "visitor_events-sessions-v2",
"interval": "2020-05-01/2020-06-01"
}
},
"tuningConfig": {
"type": "index_parallel"
}
}
}
```
the `visitor_id_sketch` is filled with nulls even though `visitor_id` dimension exists in `visitor_events-sessions-v2`. However If I explcitily add:
```
"dimensions": ["site_id", "environment_name", "visitor_id"],
"metrics": ["count"]
```
to inputSource then visitor_id_sketch is correctly filled.

This was discussed in slack in https://the-asf.slack.com/archives/CJ8D1JTB8/p1590075877215900 and I was asked to create a bug report.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the provided index_parallel ingestion spec against the affected version, comparing inputSource behavior with and without explicitly listed dimensions. Trace how dimensionsSpec and the Druid inputSource determine available dimensions, then add a regression test for visitor_id_sketch. Done means visitor_id is populated from the input datasource without being repeated explicitly.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.