Graylog2 / Graylog2/graylog2-server

start inputs delayed after server start

Open
#5,334 0 comments 0 reactions 0 assignees View on GitHub
feature triaged
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

## Current Behavior
All inputs on a Graylog Server are started milliseconds after the Web interface is available.

```
2018-11-30T12:46:22.829+01:00 INFO [JerseyService] Enabling CORS for HTTP endpoint
2018-11-30T12:46:53.785+01:00 INFO [NetworkListener] Started listener bound to [127.0.0.1:9000]
2018-11-30T12:46:53.789+01:00 INFO [HttpServer] [HttpServer] Started.
2018-11-30T12:46:53.789+01:00 INFO [JerseyService] Started REST API at
2018-11-30T12:46:53.790+01:00 INFO [JerseyService] Started Web Interface at
2018-11-30T12:46:53.791+01:00 INFO [ServiceManagerListener] Services are healthy
2018-11-30T12:46:53.806+01:00 INFO [InputSetupService] Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Running [LB:ALIVE]
2018-11-30T12:46:53.818+01:00 INFO [ServerBootstrap] Services started, startup times in ms: {BufferSynchronizerService [RUNNING]=100, OutputSetupService [RUNNING]=104, KafkaJournal [RUNNING]=224, InputSetupService [RUNNING]=414, ConfigurationEtagService [RUNNING]=524, JournalReader [RUNNING]=566, StreamCacheService [RUNNING]=781, PeriodicalsService [RUNNING]=994, LookupTableService [RUNNING]=1874, JerseyService [RUNNING]=34103}
2018-11-30T12:46:53.841+01:00 INFO [ServerBootstrap] Graylog server up and running.
2018-11-30T12:46:53.942+01:00 INFO [InputStateListener] Input [Random HTTP message generator/5c0122509854af306949cd29] is now STARTING
2018-11-30T12:46:53.967+01:00 INFO [InputStateListener] Input [Random HTTP message generator/5c0122509854af306949cd29] is now RUNNING
```

Together with the bug https://github.com/Graylog2/graylog2-server/issues/967 that the state is not persistent you could run into a situation where Graylog gets a massive overrun.

Users will connect to the Webinterface (because a loadbalancer will route them to the now _alive_ host).

All Inputs are up and might consume high amount of messages or get a massive backlog of messages from collectors that active queue.

## Possible Solution
Make input state persistent (fix: https://github.com/Graylog2/graylog2-server/issues/967 ) and add a delay for enabling loadbalancer _alive_ and a delay to activate the inputs.

## Context
Specially for unexperient users the above behaviour can be painful and might make it hard for them to recover their Graylog. It might even need iptables or other firewalls to let Graylog first start and then slowly open the gates again.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.