JSON logging can't be ingested into Elasticsearch
- Dominant language
- Java
- Stars
- 14.9k
- Forks
- 3.5k
- Avg merge
- 19h 14m
- Merged PRs (30d)
- 63
Description
Using Logstash 5.4.3:
echo '{"foo":1}' | logstash-5.4.3/bin/logstash -f abc.conf --log.format json
echo '{"foo":"bar"}' | logstash-5.4.3/bin/logstash -f abc.conf --log.format json
with the following config:
```
input {
stdin {
codec => json
}
}
output {
elasticsearch {
}
}
```
Produces the following JSON log
```json
{"level":"WARN","loggerName":"logstash.outputs.elasticsearch","timeMillis":1499431239669,"thread":"[main]>worker2","logEvent":{"message":"Could not index event to Elasticsearch.","status":400,"action":["index",{"_id":null,"_index":"logstash-2017.07.07","_type":"logs","_routing":null},{"metaClass":{"metaClass":{"metaClass":{"action":"[\"index\", {:_id=>nil, :_index=>\"logstash-2017.07.07\", :_type=>\"logs\", :_routing=>nil}, 2017-07-07T12:40:39.591Z localhost.localdomain %{message}]","response":{"index":{"_index":"logstash-2017.07.07","_type":"logs","_id":"AV0dEP_riOfCotFy8tMk","status":400,"error":{"type":"mapper_parsing_exception","reason":"failed to parse [foo]","caused_by":{"type":"number_format_exception","reason":"For input string: \"bar\""}}}}}}}}]}}
```
Which can't be ingested into Elasticsearch because there are mixed types in an array which is unsupported by Elasticsearch:
```
"action":["index",{"_id":null,"
```
Given the best place for all logs is Elasticsearch, this makes me sad!
Contributor guide
Assessment
This issue has not been assessed yet.