Dimension from inputSource is ignored when not explicitly defined
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 58m
- Merged PRs (30d)
- 233
Description
### Affected Version
v0.18.1
### Description
Given ingestion spec:
```
{
"type": "index_parallel",
"spec": {
"dataSchema": {
"dataSource": "visitor_events-sessions-test1",
"timestampSpec": {
"column": "timestamp",
"format": "auto"
},
"dimensionsSpec": {
"dimensions": ["site_id", "environment_name"]
},
"metricsSpec": [
{ "type": "doubleSum", "name": "count", "fieldName": "count" },
{ "type": "thetaSketch", "name": "visitor_id_sketch", "fieldName": "visitor_id" }
],
"granularitySpec": {
"segmentGranularity": "day",
"queryGranularity": "hour",
"intervals": [
"2020-05-19/2020-05-21"
]
}
},
"ioConfig": {
"type": "index_parallel",
"inputSource": {
"type": "druid",
"dataSource": "visitor_events-sessions-v2",
"interval": "2020-05-01/2020-06-01"
}
},
"tuningConfig": {
"type": "index_parallel"
}
}
}
```
the `visitor_id_sketch` is filled with nulls even though `visitor_id` dimension exists in `visitor_events-sessions-v2`. However If I explcitily add:
```
"dimensions": ["site_id", "environment_name", "visitor_id"],
"metrics": ["count"]
```
to inputSource then visitor_id_sketch is correctly filled.
This was discussed in slack in https://the-asf.slack.com/archives/CJ8D1JTB8/p1590075877215900 and I was asked to create a bug report.
Contributor guide
Research direction
Start by reproducing the provided index_parallel ingestion spec against the affected version, comparing inputSource behavior with and without explicitly listed dimensions. Trace how dimensionsSpec and the Druid inputSource determine available dimensions, then add a regression test for visitor_id_sketch. Done means visitor_id is populated from the input datasource without being repeated explicitly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100