apache / apache/druid

The timestamp column format is ignored if the column value is numeric

Open
#8,709 0 comments 0 reactions 0 assignees View on GitHub
Area - Batch Ingestion Bug
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

### Affected Version

Druid 0.16.0 and probably earlier also

### Description

If the value of the JSON is a `Number` then the format is ignored and god knows what happens instead (millis is used?).

The issue is in this line of code: https://github.com/apache/incubator-druid/blob/84598fba3b283cbfd6a5addd2602c7b12ba8c00c/core/src/main/java/org/apache/druid/java/util/common/parsers/TimestampParser.java#L129

# Repro

Try to ingest

```json
{"name":"V","time":2019102120}
{"name":"D","time":2019102121}
```

With format of `yyyyMMddHH`

Here is a ready made query:

```json
{
"type": "index",
"spec": {
"type": "index",
"ioConfig": {
"type": "index",
"firehose": {
"type": "inline",
"data": "{\"name\":\"V\",\"time\":2019102120}\n{\"name\":\"D\",\"time\":2019102121}"
}
},
"dataSchema": {
"dataSource": "sample",
"parser": {
"type": "string",
"parseSpec": {
"format": "json",
"timestampSpec": {
"column": "time",
"format": "yyyyMMddHH"
},
"dimensionsSpec": {}
}
}
}
},
"samplerConfig": {
"numRows": 500,
"timeoutMs": 15000,
"cacheKey": "8451100357df41f5ac5502381506e674"
}
}
```

Results in:

```json
{
"cacheKey": "8451100357df41f5ac5502381506e674",
"numRowsRead": 2,
"numRowsIndexed": 2,
"data": [
{
"raw": "{\"name\":\"V\",\"time\":2019102120}",
"parsed": {
"__time": 2019102120,
"name": "V"
}
},
{
"raw": "{\"name\":\"D\",\"time\":2019102121}",
"parsed": {
"__time": 2019102121,
"name": "D"
}
}
]
}
```

The format is totally ignored. If you send it as a string then all is good.

# Impact

This is strange and confusing behavior. One would intuitively expect that setting a JODA time format would cast the value to a string.

I was confused by it as was this ASF Slack user:

![image](https://user-images.githubusercontent.com/177816/67253535-e76e1380-f42c-11e9-9ecc-ed2af06cc134.png)

# Workaround

You could either change your data to represent these columns as strings or add the timestamp in the transform stage using `timestamp_parse("time_column_to_parse", 'yyyyMMddHH')`

Contributor guide

Open the contributing guide

Research direction

Start with core/src/main/java/org/apache/druid/java/util/common/parsers/TimestampParser.java at line 129 and run the provided inline ingestion query with numeric time values and format yyyyMMddHH. Compare the result with string input; done when numeric values honor the configured format and regression coverage verifies the behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, databases
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.