elastic / elastic/logstash

Incorrect Logstash Configuration for System Logs Documented

Open
#7,508 4 comments 4 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
14.9k
Forks
3.5k
Avg merge
1d 4h
Merged PRs (30d)
88

Description

Incorrect logstash pipeline configuration for system authorization logs documented at https://www.elastic.co/guide/en/logstash/current/logstash-config-for-filebeat-modules.html#parsing-system

Symptoms:
The following error and failure to index authorization log events:
[2017-06-21T10:02:32,974][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"active-syslog", :_type=>"auth", :_routing=>nil}, 2017-06-06T14:53:38.000Z hostname %{message}], :response=>{"index"=>{"_index"=>"syslog-000001", "_type"=>"auth", "_id"=>"AVzK9jiDxj20MZUklAjK", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [system.auth.user]", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:128"}}}}}

Problem:
The documented pipeline configuration creates a [system][auth] object with a [user] field and then attempts to create a [system][auth][user][add] object with various 'useradd' fields. This creates a type collision which causes elasticsearch to fail to index one of the types.

Solution:
Use a different object for 'useradd' events such as [system][auth][useradd]

Contributor guide

Open the contributing guide

Research direction

Start with the linked Logstash documentation section on parsing system logs and inspect the documented pipeline for system authorization and useradd events. Change the documented object path so these events do not collide, then verify that authorization log events can be indexed without the reported mapper parsing exception.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch
Domain
documentation
Issue type
Documentation
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.