[Centralized config mgmt] Error message when .logstash does not exist
- Dominant language
- Java
- Stars
- 14.9k
- Forks
- 3.5k
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 88
Description
*Original comment by @ycombinator:*
### Steps to reproduce
1. Run Elasticsearch, Logstash, and Kibana with x-pack.
2. Either start with a fresh install of all the above products or `DELETE .logstash` to simulate a fresh install.
3. Configure Centralized Configuration Management in Logstash:
```
xpack.management.enabled: true
xpack.management.elasticsearch.username: elastic
xpack.management.elasticsearch.password: XXXXXXXX
xpack.management.elasticsearch.url: "http://localhost:9200"
xpack.management.pipeline.id: [ "apache", "test" ]
```
4. Start up Logstash: `./bin/logstash`
### Observed behavior
The Logstash log shows errors like this every 5 seconds (or whatever `xpack.management.logstash.poll_interval` is set to in `logstash.yml`):
```
[ERROR] 2017-09-26 07:44:53.530 [Ruby-0-Thread-2: /Users/shaunak/logstash-6.0.0-rc1/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] sourceloader - Could not fetch all the sources {:exception=>LogStash::ConfigManagement::ElasticsearchSource::RemoteConfigError, :message=>"Empty configuration for pipeline_id: apache", :backtrace=>["/Users/shaunak/logstash-6.0.0-rc1/vendor/bundle/jruby/2.3.0/gems/x-pack-6.0.0.rc1-java/lib/config_management/elasticsearch_source.rb:96:in `get_pipeline'", "/Users/shaunak/logstash-6.0.0-rc1/vendor/bundle/jruby/2.3.0/gems/x-pack-6.0.0.rc1-java/lib/config_management/elasticsearch_source.rb:82:in `block in pipeline_configs'", "org/jruby/RubyArray.java:2481:in `collect'", "/Users/shaunak/logstash-6.0.0-rc1/vendor/bundle/jruby/2.3.0/gems/x-pack-6.0.0.rc1-java/lib/config_management/elasticsearch_source.rb:81:in `pipeline_configs'", "/Users/shaunak/logstash-6.0.0-rc1/logstash-core/lib/logstash/config/source_loader.rb:59:in `block in fetch'", "org/jruby/RubyArray.java:2481:in `collect'", "/Users/shaunak/logstash-6.0.0-rc1/logstash-core/lib/logstash/config/source_loader.rb:58:in `fetch'", "/Users/shaunak/logstash-6.0.0-rc1/logstash-core/lib/logstash/agent.rb:148:in `converge_state_and_update'", "/Users/shaunak/logstash-6.0.0-rc1/logstash-core/lib/logstash/agent.rb:105:in `block in execute'", "/Users/shaunak/logstash-6.0.0-rc1/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/interval.rb:18:in `interval'", "/Users/shaunak/logstash-6.0.0-rc1/logstash-core/lib/logstash/agent.rb:94:in `execute'", "/Users/shaunak/logstash-6.0.0-rc1/logstash-core/lib/logstash/runner.rb:362:in `block in execute'", "/Users/shaunak/logstash-6.0.0-rc1/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]}
```
This seems to be happening because the `.logstash` index does not exist yet. For testing purposes, if you manually create the `.logstash` index (`PUT .logstash`) these errors will go away. Normally the `.logstash` index will be created by the Centralized Config Management Kibana UI when the first pipeline is created via that UI.
### Expected behavior
I would think we should not show this error when `.logstash` does not exist. That would then match the behavior when the `.logstash` index does exist but no pipeline is configured in it yet. Currently the error is not actionable and just ends up adding noise to the logs.
Contributor guide
Research direction
Reproduce the missing-.logstash case with the Centralized Configuration Management settings shown, then inspect config_management/elasticsearch_source.rb around get_pipeline and pipeline_configs. Compare the logs with an existing empty .logstash index and trace the fetch through logstash-core/lib/logstash/config/source_loader.rb. Done means the missing index no longer emits a repeated, non-actionable error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch, java, ruby
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100