TransformSpec for firehoses appear to perform the operation twice
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 58m
- Merged PRs (30d)
- 233
Description
### Affected Version
Tested in 0.18
### Description
I am writing integration tests for transform specs and noticed that when using a transform spec with a parser, the transformation is being applied twice. See the below ingestion spec.
You can re-create this by sym-linking `/resources` to `$DRUID_CODEBASE/integration-tests/src/test/resources`
```
{
"type": "index",
"spec": {
"dataSchema": {
"dataSource": "wiki-tests-2",
"metricsSpec": [
{
"type": "count",
"name": "count"
},
{
"type": "doubleSum",
"name": "added",
"fieldName": "added"
},
{
"type": "doubleSum",
"name": "triple-added",
"fieldName": "triple-added"
},
{
"type": "doubleSum",
"name": "deleted",
"fieldName": "deleted"
},
{
"type": "doubleSum",
"name": "delta",
"fieldName": "delta"
},
{
"name": "thetaSketch",
"type": "thetaSketch",
"fieldName": "user"
},
{
"name": "quantilesDoublesSketch",
"type": "quantilesDoublesSketch",
"fieldName": "delta"
},
{
"name": "HLLSketchBuild",
"type": "HLLSketchBuild",
"fieldName": "user"
}
],
"granularitySpec": {
"segmentGranularity": "DAY",
"queryGranularity": "second",
"intervals" : [ "2013-08-31/2013-09-02" ]
},
"parser": {
"parseSpec": {
"format" : "json",
"timestampSpec": {
"column": "timestamp"
},
"dimensionsSpec": {
"dimensions": [
"page",
"language",
"user",
"unpatrolled",
"newPage",
"robot",
"anonymous",
"namespace",
"continent",
"country",
"region",
"city"
]
}
}
},
"transformSpec": {
"transforms": [
{
"type": "expression",
"name": "language",
"expression": "concat('l-', language)"
},
{
"type": "expression",
"name": "triple-added",
"expression": "added * 3"
}
]
}
},
"ioConfig": {
"type": "index",
"firehose": {
"type": "local",
"baseDir": "/resources/data/batch_index",
"filter": "wikipedia_index_data*"
}
},
"tuningConfig": {
"type": "index",
"maxRowsPerSegment": 10
}
}
}
```
Then run a query against it to see the value
```
SELECT
"language",
COUNT(*) AS "Count"
FROM "wiki-tests-2"
GROUP BY 1
ORDER BY "Count" DESC
```
The results look like
```
l-l-en 4
l-l-ja 2
l-l-ru 2
l-l-zh 2
```
However if you switch to the new format (inputSource/ inputFormat instead of Firehoses), it will perform the operation as expected.
Contributor guide
Research direction
Start by reproducing the ingestion with the supplied spec after symlinking /resources to integration-tests/src/test/resources, then run the shown SQL query and compare the firehose path with the inputSource/inputFormat path. Trace the transform-spec handling for the legacy firehose ingestion flow; done means each transform is applied once, so language values are l-en rather than l-l-en.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100