apache / apache/druid

TransformSpec for firehoses appear to perform the operation twice

Open
#9,589 0 comments 0 reactions 0 assignees View on GitHub
Area - Batch Ingestion Area - Streaming Ingestion Bug Contributions Welcome
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

### Affected Version

Tested in 0.18

### Description

I am writing integration tests for transform specs and noticed that when using a transform spec with a parser, the transformation is being applied twice. See the below ingestion spec.

You can re-create this by sym-linking `/resources` to `$DRUID_CODEBASE/integration-tests/src/test/resources`

```
{
"type": "index",
"spec": {
"dataSchema": {
"dataSource": "wiki-tests-2",
"metricsSpec": [
{
"type": "count",
"name": "count"
},
{
"type": "doubleSum",
"name": "added",
"fieldName": "added"
},
{
"type": "doubleSum",
"name": "triple-added",
"fieldName": "triple-added"
},
{
"type": "doubleSum",
"name": "deleted",
"fieldName": "deleted"
},
{
"type": "doubleSum",
"name": "delta",
"fieldName": "delta"
},
{
"name": "thetaSketch",
"type": "thetaSketch",
"fieldName": "user"
},
{
"name": "quantilesDoublesSketch",
"type": "quantilesDoublesSketch",
"fieldName": "delta"
},
{
"name": "HLLSketchBuild",
"type": "HLLSketchBuild",
"fieldName": "user"
}
],
"granularitySpec": {
"segmentGranularity": "DAY",
"queryGranularity": "second",
"intervals" : [ "2013-08-31/2013-09-02" ]
},
"parser": {
"parseSpec": {
"format" : "json",
"timestampSpec": {
"column": "timestamp"
},
"dimensionsSpec": {
"dimensions": [
"page",
"language",
"user",
"unpatrolled",
"newPage",
"robot",
"anonymous",
"namespace",
"continent",
"country",
"region",
"city"
]
}
}
},
"transformSpec": {
"transforms": [
{
"type": "expression",
"name": "language",
"expression": "concat('l-', language)"
},
{
"type": "expression",
"name": "triple-added",
"expression": "added * 3"
}
]
}
},
"ioConfig": {
"type": "index",
"firehose": {
"type": "local",
"baseDir": "/resources/data/batch_index",
"filter": "wikipedia_index_data*"
}
},
"tuningConfig": {
"type": "index",
"maxRowsPerSegment": 10
}
}
}
```
Then run a query against it to see the value

```
SELECT
"language",
COUNT(*) AS "Count"
FROM "wiki-tests-2"
GROUP BY 1
ORDER BY "Count" DESC
```
The results look like
```
l-l-en 4
l-l-ja 2
l-l-ru 2
l-l-zh 2
```
However if you switch to the new format (inputSource/ inputFormat instead of Firehoses), it will perform the operation as expected.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the ingestion with the supplied spec after symlinking /resources to integration-tests/src/test/resources, then run the shown SQL query and compare the firehose path with the inputSource/inputFormat path. Trace the transform-spec handling for the legacy firehose ingestion flow; done means each transform is applied once, so language values are l-en rather than l-l-en.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.