elastic / elastic/logstash

JSON logging can't be ingested into Elasticsearch

Open
#7,610 6 comments 5 reactions 0 assignees View on GitHub
bug enhancement modules
Dominant language
Java
Stars
14.9k
Forks
3.5k
Avg merge
19h 14m
Merged PRs (30d)
63

Description

Using Logstash 5.4.3:

echo '{"foo":1}' | logstash-5.4.3/bin/logstash -f abc.conf --log.format json
echo '{"foo":"bar"}' | logstash-5.4.3/bin/logstash -f abc.conf --log.format json

with the following config:
```
input {
stdin {
codec => json
}
}
output {
elasticsearch {
}
}
```

Produces the following JSON log

```json
{"level":"WARN","loggerName":"logstash.outputs.elasticsearch","timeMillis":1499431239669,"thread":"[main]>worker2","logEvent":{"message":"Could not index event to Elasticsearch.","status":400,"action":["index",{"_id":null,"_index":"logstash-2017.07.07","_type":"logs","_routing":null},{"metaClass":{"metaClass":{"metaClass":{"action":"[\"index\", {:_id=>nil, :_index=>\"logstash-2017.07.07\", :_type=>\"logs\", :_routing=>nil}, 2017-07-07T12:40:39.591Z localhost.localdomain %{message}]","response":{"index":{"_index":"logstash-2017.07.07","_type":"logs","_id":"AV0dEP_riOfCotFy8tMk","status":400,"error":{"type":"mapper_parsing_exception","reason":"failed to parse [foo]","caused_by":{"type":"number_format_exception","reason":"For input string: \"bar\""}}}}}}}}]}}
```
Which can't be ingested into Elasticsearch because there are mixed types in an array which is unsupported by Elasticsearch:
```
"action":["index",{"_id":null,"
```
Given the best place for all logs is Elasticsearch, this makes me sad!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.