elastic / elastic/logstash

[discuss] default plugins dependencies smoke test proposal

Open
#7,666 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
14.9k
Forks
3.5k
Avg merge
19h 14m
Merged PRs (30d)
63

Description

Relates to #7627 - [meta] streamline and harmonize plugins with jars build and publish processes

This configuration contains all the [default plugins](https://github.com/elastic/logstash/blob/4ceaa9f2e26b444758cac49ce908d4312f60e2db/rakelib/plugins-metadata.json)
```
input {
stdin { codec => collectd { typesdb => "./bin/logstash" }}
stdin { codec => dots {}}
stdin { codec => edn {}}
stdin { codec => edn_lines {}}
stdin { codec => fluent {}}
stdin { codec => es_bulk {}}
stdin { codec => graphite {}}
stdin { codec => json {}}
stdin { codec => json_lines {}}
stdin { codec => line {}}
stdin { codec => msgpack {}}
stdin { codec => multiline { pattern => "foo" what => "next" }}
stdin { codec => netflow {}}
stdin { codec => plain {}}
stdin { codec => rubydebug {}}

heartbeat {}
#dead_letter_queue {} # fails with missing jar
couchdb_changes { db => "foo" }
elasticsearch {}
exec { command => "" interval => "1" }
file { path => "foo" }
ganglia {}
gelf {}
generator {}
graphite { port => "666" }
http {}
http_poller { urls => { "foo" => "bar" }}
imap { host => "foo" user => "bar" password => "baz" }
irc { host => "foo" channels => "bar" }
jdbc { jdbc_driver_class => "foo" jdbc_connection_string => "bar" jdbc_user => "baz" }
log4j {}
lumberjack { port => 666 ssl_certificate => "./bin/logstash" ssl_key => "./bin/logstash" }
pipe { command => "foo" }
rabbitmq { host => "foo" }
redis { key => "foo" data_type => "list" }
s3 { bucket => "foo" }
snmptrap {}
sqs { queue => "foo" }
stdin {}
syslog {}
tcp { port => 666 }
twitter { consumer_key => "foo" consumer_secret => "bar" oauth_token => "baz" oauth_token_secret => "biz" }
udp { port => 666 }
unix { path => "foo" }
xmpp { user => "foo" password => "bar" }
kafka {}
beats { port => 666 }
}

filter {
clone {}
csv {}
date { match => [ "foo", "M" ]}
dissect {}
dns {}
drop {}
fingerprint {}
geoip { source => "foo" }
grok {}
json { source => "foo" }
kv {}
metrics {}
mutate {}
ruby { code => "" }
sleep {}
split {}
syslog_pri {}
throttle { key => "foo" }
urldecode {}
useragent { source => "foo" }
uuid { target => "foo" }
xml { source => "foo" }
}

output {
xmpp { user => "foo" password => "bar" message => "baz" }
cloudwatch {}
csv { path => "foo" fields => "bar" }
elasticsearch {}
file { path => "foo" }
graphite {}
http { url => "foo" http_method => "post" }
irc { host => "foo" channels => "bar" }
kafka { topic_id => "foo" }
nagios {}
null {}
pagerduty { service_key => "foo" }
pipe { command => "foo" }
rabbitmq { host => "foo" exchange => "bar" exchange_type => "direct" }
redis {}
s3 { bucket => "foo" }
sns {}
sqs { queue => "foo" }
statsd {}
stdout {}
tcp { host => "foo" port => 666 }
udp { host => "foo" port => 666 }
webhdfs { host => "foo" user => "bar" path => "baz" }
}
```

and can be used with this command to test the loading/initialization and dependencies loading (including jar files) with

```sh
bin/logstash -t -f default-plugins.conf
```

If all plugins are successfully loaded, then logstash exits with a shell exit status of `0` and this output
```
Configuration OK
[2017-07-12T13:38:10,286][INFO ][logstash.runner ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash
```

Otherwise it will exit with and error log and a shell exit status of 1`, for example by uncommenting the `dead_letter_queue` input in the above config

```
[2017-07-12T13:37:13,765][ERROR][logstash.plugins.registry] Problems loading a plugin with {:type=>"input", :name=>"dead_letter_queue", :path=>"logstash/inputs/dead_letter_queue", :error_message=>"NameError", :error_class=>NameError, :error_backtrace=>["/Users/colin/dev/src/elasticsearch/logstash/logstash-core/lib/logstash/plugins/registry.rb:226:in `namespace_lookup'", "/Users/colin/dev/src/elasticsearch/logstash/logstash-core/lib/logstash/plugins/registry.rb:162:in `legacy_lookup'", "/Users/colin/dev/src/elasticsearch/logstash/logstash-core/lib/logstash/plugins/registry.rb:138:in `lookup'", "/Users/colin/dev/src/elasticsearch/logstash/logstash-core/lib/logstash/plugins/registry.rb:180:in `lookup_pipeline_plugin'", "/Users/colin/dev/src/elasticsearch/logstash/logstash-core/lib/logstash/plugin.rb:140:in `lookup'", "/Users/colin/dev/src/elasticsearch/logstash/logstash-core/lib/logstash/pipeline.rb:125:in `plugin'", "(eval):72:in `'", "org/jruby/RubyKernel.java:1000:in `eval'", "/Users/colin/dev/src/elasticsearch/logstash/logstash-core/lib/logstash/pipeline.rb:82:in `initialize'", "/Users/colin/dev/src/elasticsearch/logstash/logstash-core/lib/logstash/runner.rb:320:in `block in execute'", "org/jruby/RubyArray.java:1734:in `each'", "/Users/colin/dev/src/elasticsearch/logstash/logstash-core/lib/logstash/runner.rb:319:in `execute'", "/Users/colin/dev/src/elasticsearch/logstash/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "/Users/colin/dev/src/elasticsearch/logstash/logstash-core/lib/logstash/runner.rb:219:in `run'", "/Users/colin/dev/src/elasticsearch/logstash/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "/Users/colin/dev/src/elasticsearch/logstash/lib/bootstrap/environment.rb:71:in `'"]}
[2017-07-12T13:37:13,769][FATAL][logstash.runner ] The given configuration is invalid. Reason: Couldn't find any input plugin named 'dead_letter_queue'. Are you sure this is correct? Trying to load the dead_letter_queue input plugin resulted in this error: Problems loading the requested plugin named dead_letter_queue of type input. Error: NameError NameError
```

## Proposal ##

- [ ] add a new field in [logstash/rakelib/plugins-metadata.json](https://github.com/elastic/logstash/blob/4ceaa9f2e26b444758cac49ce908d4312f60e2db/rakelib/plugins-metadata.json) named `sample-config: "..."` which will contain a string of a syntax-valid plugin config, basically the plugin configs that are required as above in the default plugins config file.
- [ ] create a task to generate the default plugins config file based on `logstash/rakelib/plugins-metadata.json` for the `"default-plugins": true` and using the `sample-config: "..."` field.
- [ ] add a package generation smoke test to execute `bin/logstash -t -f default-plugins.conf` and check for exit status

Note: the only tricky config of all default plugins is with the `lumberjack` input and `collectd` codec which actually verifies the existence of the configured files. We can decide how to deal with it, but for now I just tricked it with a file I know is part of the package :)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.