apache / apache/druid

Transform specs are ignored if dimension auto detection is used

Open
#7,952 0 comments 0 reactions 0 assignees View on GitHub
Area - Batch Ingestion Area - Web Console
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

The column auto detecting feature does not find the new columns created by transforms.

### Affected Version

All versions of Druid so far (up to 0.15.0)

### Description

Say you have data:
```
{"a":"hello","b":"world"}
{"a":"where","c":"to go"}
```

In a file that lives at: `/Users/vadim/Downloads/test-data.json`

And you ingest it with:
```
{
"dataSchema": {
"dataSource": "Downloads",
"parser": {
"type": "string",
"parseSpec": {
"format": "json",
"timestampSpec": {
"column": "!!!_no_such_column_!!!",
"missingValue": "2010-01-01T00:00:00Z"
},
"dimensionsSpec": {}
}
},
"metricsSpec": [
{
"name": "count",
"type": "count"
}
],
"granularitySpec": {
"type": "uniform",
"segmentGranularity": "DAY",
"queryGranularity": "HOUR",
"rollup": true,
"intervals": null
},
"transformSpec": {
"filter": null,
"transforms": [
{
"type": "expression",
"name": "a_prime",
"expression": "concat(\"a\",'_prime')"
}
]
}
},
"ioConfig": {
"type": "index_parallel",
"firehose": {
"type": "local",
"baseDir": "/Users/vadim/Downloads",
"filter": "test-data.json"
},
"appendToExisting": false
},
"tuningConfig": {
"type": "index_parallel",
"maxRowsPerSegment": null,
"maxRowsInMemory": 1000000,
"maxBytesInMemory": 0,
"maxTotalRows": null,
"numShards": null,
"indexSpec": {
"bitmap": {
"type": "concise"
},
"dimensionCompression": "lz4",
"metricCompression": "lz4",
"longEncoding": "longs"
},
"maxPendingPersists": 0,
"forceGuaranteedRollup": false,
"reportParseExceptions": false,
"pushTimeout": 0,
"segmentWriteOutMediumFactory": null,
"maxNumSubTasks": 1,
"maxRetry": 3,
"taskStatusCheckPeriodMs": 1000,
"chatHandlerTimeout": "PT10S",
"chatHandlerNumRetries": 5,
"logParseExceptions": false,
"maxParseExceptions": 2147483647,
"maxSavedParseExceptions": 0,
"partitionDimensions": [],
"buildV9Directly": true
},
"type": "index_parallel"
}
```

Notice how I am trying to create an `a_prime` column with a transform spec.

The job will work but when you query the data:

![image](https://user-images.githubusercontent.com/177816/60033822-28f8d880-965e-11e9-9cac-557fcc714b87.png)

You see that there is no `a_prime` column.

I would be great (and would make a ton more sense) if the transforms added themselves to the column list coming from the file.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the supplied index_parallel ingestion spec with an empty dimensionsSpec and the expression transform that creates a_prime. Trace how column auto detection builds its dimension list relative to the transformSpec. Done means transformed columns such as a_prime appear in the ingested data and are queryable when dimensions are auto detected.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.